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Abstract 

The major aim of this project was to develop dissemination packages 
for reading programs that had demonstrated ef tectiveness in improving read- 
ing achievement. Up to /'j packaqes were to be developed. In addition, a 
catalog of reading programs was to be prepared. The total effort can be 
regarded as one whose intended objective ir, to introduce change through the 
dissemination of information about effective reading programs and practices. 

AIR recommendations for packaging and c ir.aloging were based on an 
intensive nationwide search for program caticliciates which were then screened 
on the basis of program description and evaluation information. The search 
involved review of past research studies, computer-stored abstracts, and 
library materials, as well as the obtaining of nominations from experts in 
the field and staff of educational, professional, and government organiza- 
tions. Program information was obtained from a study questionnaire, the 
Program Information Form (PIF), and from more detailed program documents. 
Initial screening, using a computer, scored and rank-ordered reading pro- 
grams on criteria of effectiveness and on adequacy of program evaluation. 
Screening was iterative with successively more stringent standards applied 
at each stage; senior research staff conducted the more extensive examina- 
tion of available program evidence. 

The results of the search and screen tasks were as follows: over 1500 
program candidates identified through nominations and literature searches 
were sent PIFs; 728 of these returned com|)leted PIFs which were key punched 
and computer scored to rank programs on the basis of preceded answers to 
items pertaining to program evaluation (i.e., concerning criterion measures, 
statistical adequacy, experimental design, and other claims or considera- 
tions); 27 programs were recommended for packa^jinq and 222 were selected 
for the catalog. The 27 recommended for j'.v.kc <;i ng were a very small frac- 
tion of the programs reviewed, but they were the only candidates judged by 
AIR staff reviewer?; to have met defensible standards for claims of effec- 
tiveness. Even this group contained programs that compromised evaluation 
rigor. 

One of five exemplary reading programs previously packaged by Right to Read 
was approved by the Office of Education but was not repackaged; also six programs 
selected in a concurrent Title I packaging project sponsored by the Office of 
Planning, Budgeting, and Evaluation were not candidates for AIR selection. 

The search and screen activities described above fell within AIR's 
contract with Richt to Read. However, during the search and screen and 
prior to Right to Read approval of AIR recommendations, an additional 
screening stage \/a~. required by the Office of rdjcn u' on Dissemination 
Review Panel (DRP). The PRP reviewed program description and evaluation 
information prepared by AIR for the 27 programs recommended for packaging. 
DRP approved 14 of these for dissemination and action is pending on 2. Of 
the 14, Right to Read approved 12 for packaging. 

a 



Package development involved 5-day site visits to collect program • 
information for a filmstrip and sound cassette, a management handbook, 
an instructional handbook, and charts that summarized program processes 
and related objectives-actiyities-and-assessments . Information in the 
filmstrip provides viewers with program highlights and simul ated expcsure 
to classroom activities. The more detailed information in the management 
and instructional handbooks helps administrators at a new cite judge whether 
the program is suitable for local impU.i.ontation. The handbooks are also 
useful for inservicG training of managers, instructors, and other program 
personnel, and as tools for them to ust. v.i'Sn the program is implemented. 
The two charts have similar function'^, buinmarizing in capsule form the most 
essential features of the program. Intended mainly for program managers and 
teachers, the packages were developed to serve a variety of purposes and to 
be appropriate for a wider audience of persons concerned with educational 
programs (e.g., school boards, parents, community members). A guiderule 
was also produced that summarized 17 items of comparative information for 
the 12 packaged programs. 

The catalog of reading programs contains 222 one-page summaries of 
basic information under the following standard headings. Program Size and 
Target Population; Year Started; Staff; Major Features; Facilities, Materials, 
Equipment; Cost; and For Further Information. The use of standard headings 
and the separation of program summaries into elementary, secondary, adult, 
and special education sections of the catalog, enables the reader to quickly 
compare information across programs. Taken together, these program summaries 
represent a wide range of approaches to teaching reading and provide a wealth 
of ideas that may be adapted to fit local objectives and needs. 

Looking to the future, MR included two sets of recommendations in the 
final report. The first set pertains to the need for upgrading the quality 
of local program evaluation and includes specific suggestions for improve- 
ment. The second set pertains to plans for disseminating the packaged pro- 
grams. AIR recommends that a comprehensive diffusion and implementation 
plan be developed so that the exemplary reading programs packaged under this 
contract can be successfully implemented in new locations. A delivery 
system must be specified which brings together potential users, packaged 
programs, and change agents who can supply whatever training and implementa- 
tion support new sites need, when they need it. In making this recommenda- 
tion, AIR assumes that the packages alone will not cause anything new to 
happen. Instead, they must be activated by other components in a coordinated 
plan to help new sites select, adapt, implement, and evaluate these out- 
standing reading proofams. A separate technical report, summarized below, 
was prepared on these aspects of diffusion planning. 

The supplement to the final report, entitled Diffusion Pl anning f or the 
Rig ht t o Read PackJ9ec' Pj^oqrams, was prepared for "Right to Read to use in 
developing an e-fvective plan for interesting potential usr.rs in examining and 
possibly adopting *-.he validated programs packaged under this contract. The 
six chapters in the document address key steps in the diffusion process: 
setting objectives \or the diffusion effort, selecting sites where the packaged 
programs will succeed, netting information about these programs to potential 
users, finding out wf^a' >itej need tc implement them, supplying what sites need 
when they need it, aiiJ evaluating the effectiveness of the diffusion effort. 
Each chapter integrates intormation obtained from field interviews and from 
selected current litt "etjre. This information comes from implementers and 
researchers who have introduced or studied educational change. Based on these 
experiences, impl icatio-',;. for use of the Right to Read packages are listed at 
the end of each chapter to provide specifications for developing a sound dif- 
fusion strategy--wheth(,r I'or a field test of the packaged progrdms or for a 
limited disseminatiori effort. a 
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Chapter I 

INTRODUCTION AND OVERVIEW OF THE PROJECT 



Bac kground 

This is the final report of a project, spoir.ored by the National Riqht 
to Read Program of the Office of Education, U.S. Department of Health, 
Education, and Welfare, under Contract OEC-0-73-7054, conducted by the Ameri- 
can Institutes for Research (AIR) entitled "Identifying, Validating, and 
Multi-Media Packaging of Effective Reading Programs." The title describes 
three central tasks of the project: (1) the search for programs that were 
effectively improving reading instruction in the United States, (2) the 
refinement of criteria and the development of instruments and procedures to 
review and to screen reading programs on the basis of these effectiveness 
criteria, and (3) the development of multi-media products that describe 
reading programs in a manner conducive to widespread dissemination. A 
fourth task, one that is more often implicit than realized in similar efforts, 
was the review of diffusion strategies so that the products of the project 
might be effectively brought to the attention of potential users for their 
examination and possible adoption. 

Central to these tasks Wcis the definition of a reading program. As 
defined for the purposes of this project, a program is viewed as an instruc- 
tional system ongoing at a specific location that is describable in terms of 
components or elements. These include needs, objectives, contexts, con- 
straints, curricula, staffing organizations and policies, participant char- 
acteristics, facilities and plant, management plans, costs and budget, and 
procedures for evaluation. Materials of instruction, therefore, represent 
only one of the program components as defined here, rather than a "reading 
program" as often conceptualized by a publisher. This definition of a 
program certainly involves materials such as texts, books, coordinated 
tests and exercises, but for the purposes of this project these are viewed 
as materials used by teacfiers in a wider learning context. The context in 
which learning occurs, regardless of curriculum or theory, involves stimuli, 
motivators, active subjects, reinforcement schedules, and evaluation procedures. 

The Right to Read-AIR project is basically an information-processing 
effort. The quantity of raw information about reading programs in the 
United States is not only enormous, but is available in a variety of formats 

ERIC ^ 



from many sources, and exists in differing levels of quality. Locating and 
obtaining this information was the search task. The screen-and-review tasks 
reorganized information obtained about reading programs into a common basis 
that permitted cross-program comparisons, that was objectively measurable, 
that was meaningful in a policy sense both to the sponsor and to the educa- 
tional community, and that allowed educators and practitioners to relate 
program information to management, instructional, and evaluation objectives 
and operations. Screening criteria were operational i zed through instruments 
and review procedures that sifted information for each program and finally 
orderec all reviewed programs along common dimensions of program effectiveness. 

Although the search and screen tasks were concerned with information 
retrieval and processing, the project's main task was the preparation of 
program descriptions in order to disseminate information about effective 
programs. Dissemination is tied to the transformation and delivery of re- 
structured information to potential users who in most instances share similar 
roles with the original instructional program developers. Thus, the entire 
project can be overviewed as one directed toward a diffusion goal whose in- 
tended effect is to "i '.troducc the possibility of che-^ge th-ough the dissemi- 
nation of information about effective reading programs and practices. 

The two major project products are a set of reading program packag«?s 
and a catalog of reading program descriptions. Packages were developed for 
programs that were judged through AIR screening procedures to be effective 
and that were also subsequently approved by the Office of Education for dis- 
semination as exemplary programs. As in all selection systems, hits and 
misses occurred. Procedures, notably a sequencing of increasingly tighter 
reviews, were designed to minimize both of these kinds of errors. 

The Search 

Chapter II details the search for effective programs which attempted 
to spread as wide a net as possible in order not to Overlook effective pro- 
grams. Contacts with professionals included sending an approved Program 
Nomination Form to a panel of nearly 100 educators, teachers, evaluators, 
reading theorists, educational sociologists, media experts, publishers, and 
government officials. Program Nomination Forms were also sent to State 
Departments of Education, School Superintendents of cities with populations 
of 100,000 or higher, research directors, and numerous professional organiza- 
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tions. All toqetlier, fjCiOO 'loiiiitidtion forms were mailed to these serTch 
sources (over 1300 addre'^sees ) . In one respect, the search it^.elf became 
tha initial screening review, since noiiiinatorr> were asked to identify loca- 
tions wh^re reading inr.tr^uction was taking place, although the question of 
whether or not the instruction was effective was deliberately ignored at 
this step. From the completed forms which were returned, 963 candidate 
programs were identified. The National Right to Read Program supplied list^, 
and materials that yielded leads to 320 progr-ams, and a national advctisinq 
campaign brought in another 65 nominations. Literature searches identified 
about 375 candidate programs. 

In summary, over 1700 leads to candidate programs were retained through 
the various search strategies. Checks for duplication reduced the randida e 
pool to 1 520 programs . 

Criteria of Effectiveness 

Chapter III describes the criteria defined to process the iniormation 
obtained from surveyed reading programs. Criteria definitions were based 
upoii thosi listed in the Right to Road request for proposal (RFP). Ihese 
criteri?. needed to be operational i zed to allow comparisons of readirq pro- 
grams with one another on common bases so that the\' could he scaled in 
terms of their relative effectiveness. 

Program criteria for screenin., included; (1) location withiri tne 
United States, (2) operation for at least 1 year and the expectation of 
2 addil.ional years of 0()eratinn, (3) availability of program evaluation 
(-.■vid-^nce reported sii^ce 1968. (4) a focus on reliabl> iv'^-^isured reading 
achievement, (5) an adequate assessment design and statistical treatment 
comparing reading achievement gains for program participants wi Ui (i.iins for 
a credible non-participanc group, (6) adequate size, and (7) potential for 
replication. These criteria indicate that adequate local evaluation was a 
necessary, though not sufficient, condition for programs to be screened for 
dissemination. Ultimately, however, tiie quality of local program evaluation 
wa^^ the major discriminator between effective and non-effccti vc prigrims. 
This condition sorted programs into the majority set characterised l)y faulty 
local '.'Valuations and the relatively few who survived close examination, 
both by i.he AIR staff and finally by the Dissemination Review Panel of the 
Office of Education. 
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The quality of local evaluation became essentially synonymous with the 
judged quality of the program, because relatively few programs reported eval- 
uations that were sufficiently sound to demonstrate conclusive interpreta- 
tions regarding learner outcomes. One of the strong recommendations result- 
ing from this study is that lical evaluations must be upgraded. Evaluation 
should be conceived of as a management function that begins with the planning 
and start of the total instructional program. It is only through sound local 
evaluation designs that program managers and instructors can develop necessary 
feedback data for program improvement and can begin to determine which program 
components or elements require modification. 

Instr u mentation 

Chapter III also describes the instrument and procedures developed to 
process the information obtained from candidate reading programs. The 
major instrument used for assessing program effecti venebs was the Program 
Information Form (PIF). The PIF contained precoded, objectively scored 
items and alternatives as well as open-ended items. This format enabled 
programs to be initially scored on effectiveness with the use of a computer, 
and to be examined in greater depth by AIR staff review of available program 
description and evaluation documents. 

A PIF was sent to each program candidate located in the search with a 

request for local program descriptions and evaluations. During the project, 

1520 PIFs were sent to reading programs of which 804 were returned to AIR. 

Nearly 100 of these were incomplete; 728 were submitted to in-house data 
processing. 

The Screening Process 

Chapter IV reports the procedures for screening and selecting programs 
to be packaged and procedures for selecting programs to be described in the 
catalog. These procedures define the interface of information collection 
and information delivery. 

Information obtained from the .responses to 55 of tlie PIF items was 
summarized into six subscales and a total weighted (composite) scale. 
Scores on the composite scale nntially rank-ordered the 728 programs who 
supplied PIF information in terms of overall quality with respect to the 
screening criteria. Following intensive examination of the programs 
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scoring highest in the initial rank-ordering, only 26 programs could be 
recommended by AIR for packaging, a small fraction of those examined. Of 
other programs located and reviewed later, an additional program was recom- 
mended, bringing the total to 27. A reliability check relating the compos- 
ite score rank-order to overall quality of program evaluation by four AIR 
senior staff raters showed acceptable agreement. Subsequent analysis also 
showed agreement between the composite scale score and the approved or 
disapproved decisions by the Dissemination Review Panel. 

Reasons for Rejection 

Chapter IV also discusses the results of the screening procedures and 
focuses upon the reasons that programs failed the screening process and 
therefore were not recommended for packaging. Rn commendations for the 
improvement of local evaluation efforts conclude Chapter IV. 

P roduct Development 

The two major products of this project, discussed in Chapter V, are 
(1) a set of packages presenting multi-media descriptions of effective 
rviading programs and (2) a catalog of reading program descriptions. Since 
the purpose of preparing package descriptions for effective programs is to 
distribute information about them in a form that permits and fosters repli- 
cation, it is obvious that the dissemination material must be conceptually 
organized within the context of a diffusion plan. 

Each program package was planned to contain a filmstrip with accompany- 
ing audio commentary, a management handbook, instructional handbook, and 
flow charts. The basis for package development assumed that: 

• Programs to be packaged were those judged to be effective. 

• Descriptions were to be faithful to the program. 

• Descriptive information was directed both toward educational 
managers such as superintendents, principals, and project directors, 
and toward teachers and other instructional staff. 

• The same information was to be presented in different levels of 
detail. A filmstrip, which cannot carry a heavy information load, 
was planned to highlight each program's essential qualities, to 
enable the viewer to see classroom procedures, and to motivate 



viewers interested in replication to examine the more detailed 
descriptions provided in the handbooks for teachers and managers. 



A guiderule, summarizing 17 items of information for the 12 packaged 
programs, was also produced. This circular rule permits one to quickly and 
conveniently compare the programs on a variety of characteristics. 

The catalog of reading program descriptions was planned tc present 
summaries of the unpackaged programs that were rated highest or the initial 
PIF rank-ordering of effectiveness. Programs selected for the catalog were 
representative of different age levels and different types of participant 
groups, both in school and out, and reflected at least attempts to develop 
evaluations of program effectiveness. These programs, unlike those that 
were packaged, were not officially endorsed by the Office of Education for 
dissemination as effective programs. 

Chapter V concludes with a discussion and general outline of a diffusion 
plan for disseminating, implementing, and utilizing the products from this 
project. The basic assumption is that it is impossible to design a sponta- 
neously replicable information package. Rather, the packages -ind the cata- 
log must be regarded as components of a diffusion plan that includes 
making potential users aware of these materials, coordinating their use with 
potential sponsors and adopters, possibly preparing special training materials 
for workshops and teacher training sessions, identifying key opinion leaders 
for implementation, planning implementation trials, and assisting in the 
development of an evaluation plan for installation and program modification 
prior to full-scale adoption. 
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Chapter II 

THE SEARCH TOR EFFECTIVE READING PROGRAMS 

Overv iew of Procedures 

Rinht to Read's aim for the study was that a nationwide search bo made 
for outstandinq readinq proqrams vsith sound evidence} of their effectiveness. 
AIR was to look for any hinhly effective readinn nroqram, for any aqo or 
tarqet nonulation, funded by any i:ieans, sponsored by any institution or 
qroup. To scan such a vast array of uossi bi 1 i ties , several systematic 
methods and special instruments were develoned. In this section, the 
methods developed and used to identify leads to candidate n>^oqrams are 
described in detail. 

Durinq the 5-month period between August and December 1973, about 
1500 candidate proqrams were identified as a result of a very extensive 
search for effective readinq nroqrams. The two main methods used to locate 
candidate proqrams were nominations from the field and a literature search. 
The nomination procedure souqht leads to outstandinq readinq nroqrams known 
to members of the study's Proqram Nomination Panel, officials of federal, 
state, and local educational aqencies, members of nrcfes'-.innal ornani nations , 
ar;.i other concerned persons who read notices of the nationally advertised 
study and wished to nominate their own or other proqrams as candidates. The 
literature review consisted mainly of two comnuter searches and a subsequent 
examination of microfiche copies of about 350 of the documents thus identified. 
In addition, suitable candidates for the study were souqht in nearly one dozen 
major previous searches for successful .proqrams. A description of these and 
other search sources is provided in the remainder of this section. 

Prograiii Nomination Form 

AIR staff developed a special instrument, called the Proqram Noiiiination 
Form (PNF), to use in the search for candidate oroorams . Several cover 
letters appropriate for the nroup solicited for nominations were also devel- 
oped. A --^heet listinq and defining the criteria for elinible proqrams was 
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prepared to enclose with the cover letter and the PNFs. These materials are 
described below and are included in Appendix A. 

The PNF was printed on two sides of one sheet of paper. The form asked 
for basic identification and location inforinacion for the program beinn 
nominated as an outstanding reading project, practice, or approach. The 
nominator was also asked to indicate why he felt the pronram vas exemnlary. 
and to indicate the nature and source of evidence to supoort his clioice. 
These questions were asked to increase the likelihood that the nominator 
would give careful thought to his choices. Generally speaking, the PNFs 
were mailed to secondary sources, i.e., to people who might know of pronraiiis 
but were not directly involved in their implementation. Several copies o'" 
the PNF were sent with each cover letter. 

The cover letters that accompanied the PNFs were appropriately modified 
versions designed to communicate information about the study to the various 
sources described below. Each letter explained the purpose of the study and 
the extent of participation requested of nominators in providing leads to 
reading programs. The nominators were asked to consider community-based as 
well as school-based programs. The cover letters indicated that AIR would 
screen and validate nominated programs; this information was supplied in the 
hope that it would encourage nominators to weigh carefully the relative merits 
of alternative nominations. 

The accompanying sheet listing and defining the criteria which should 
be considered in nominating reading programs supplied broad guidelines for 
the nominators' selections. These criteria indicated that to be nominated > 
reading programs must have been in operation for at least one year, must be 
focused primarily on improving the reading achievement of participants, and 
must make available recent evaluation data (since 1 Jariuary 1968). It was 
also indicated that, ideally, some evaluation data showing the program's 
success in improving the reading ability o*" participants was desirable. The 
reference to evaluation data was deliberately subdued in favor of encour- 
aging nominators to suggest programs they regarded as successful and worthy 
of dissemination. Nominators were not asked to rate. programs on these cri- 
teria because it would have been quite inappropriate to rely on the possibly 
inaccurate or hazy information available to a nominator in screening programs. 
Also, lack of comparability among nominators in applying rating standards 
would have made this procedure inadvisable. 
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Each group that was asked to nominate readinn proqrams for the study 
is described below. 

Pr ogram Nom ination Panel. A list of paiiel niGii.bers is presentod in 
Appendix B. The Program Nomination Panel was an interdisciplinary oroup 
formed of nearly 100 members who were experts from several fields, includino 
readinq, linguistics, special education, learninq disabilities, educational 
research, educational television and media, educational socioloqy, cultural 
anthropology, compensatory education, publishing, and government. The mombers 
were selected by AIR in consultation with both the internal and external ad- 
visory panels for this study. Additional members wero identified from the 
files of National Assessment of Educational Progress (NAEP) reviewers. Final- 
ly, membership was adjusted to comply with recommendations of the Project 
Officer and to make the grouD broadly representative with respect to nro- 
fessional affiliation or institution, ethnicity, specialty, sex, and neo- 
graphical location. Geoqraohical representation on the panel was roughly 
proportional to the number of children attending public and private elemen- 
tary and secondary schools in each of the nine U.S. Office of Education 
geographical regions. Each panel member was mailed a cover letter, the 
criteria sheet, and several PNFs. Nearly 300 PNFs were mailed to this 
group* About half of the members of the panel supplied nominations, and on 
the basis of their completed PNFs. 46 candidate programs were identified, 

Federal agencies, bureaus, regi onal educational laboratories, research 
and development laboratories, and the National Right to Read Program . Partic- 
ularly good sources of program leads were provided by the National Rinht to 
Read Program and other personnel within the Office of Education in the form 
of lists, published series of project summaries, letters from project direc- 
tors, and so on. These materials were carefully examined, with a special 
effort made to identify community-based reading programs. Only solid leads 
to programs were considered, and an attempt was made to net adequate address 
information when this was not provided. On the other hand, urogram proposals 
with no indication that the proposal had been funded or implemented were 
not followed up. These materials supplied titles and address information 
for 320 projects; as these were assumed to be suitable candidates, Proqram 
Information Forms were mailed to each. (The Program Information Form is 
described elsewhere in this report.) 
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The routine for all other federal sources indicated in the heading 
above was to mail a cover letter explainina the project, the sheet listinq . 
and explaining the study's criteria, and several PNFs. Over 300 PNFs were 
mailed to this group; of this number, over 200 went to community-based Right 
to Read project staff so that they could provide nominations for more programs 
in the private sector. 

On the basis of completed PNFs returned by these sources, 29 candidate 
programs were identified. 

State Departments of Education Superintendents, Reading Supervisors, and 
Right to Read Coordinators . State Superintendents of Instruction were noti- 
fied of the study, and their assistance in securing leads to candidate pro- 
grams was requested. The cover letter in this case also included a query con- 
cerning the existence of state-wide test data that might indicate schools 
where good reading programs vyere operating. Reading supervisors and Riaht 
to Read Coordinators in each state department of education were also asked 
to nominate candidate programs. Nearly 500 PNFs were mailed to these sources 
and from those that were returned, 171 candidate programs were identified. 

Local education agencies . Cover letters, criteria, and over 500 PNFs 
were mailed to 182 school superintendents in cities with populations of 100,000 
or higher. The superintendents were asked to use the PNFs if they knew of 
good reading programs for adults, handicapped persons, institutionalized 
participants, or other selected groups which are not typically included in 
searches of this type. From the completed PNFs returned by these local 
education agencies, an additional 128 candidate programs were identified. 

Professional organizations and other groups . Nearly 4000 PNFs were 
mailed to state affiliates of national professional organizations and to 
other groups which might be promising sources of leads to exemplary reading 
projects. A complete list of these groups is contained in Appendix C. Many 
of these groups were contacted as part of AIR's intensified search for adult 
programs, special education programs, and programs ooerating in the commun- 
ity rather than in school settings. Publishers, contacted individually and 
through the President of the American Association of Publishers, were treated 
as any other nomination source. That is, thev were asked to use the PNF to 
indicate a specific site where their program or materials were being used 
with outstanding success. The other national professional groups included 
organizations of teachers, administrators, school board officials, univer- 
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sity extension personnel, black professionals, readinn specialists, and so 
on. All together, the PNFs returned by members of these organizations and 
groups provided 589 candidate reading proaranis for the study. 

Special additional e ffort t o locate adul t__an^ spe cial education nronrains . 
Although the cover letters asked for nominations for community-based nronrfims 
as well as those operating in schools, very few nominations for the former were 
received. With the assistance of Right to Redd, additional groups thought to 
be good sources for' such nominations were contacted. These groups are included 
in the lift in Appendix C. Also, organizations that had not responded to ini- 
tial contacts were recontacted and urged to send in these riominetions . 

Published announcement s of th e sear ch. Professional organizations were 
also provided with advertising copy to use for a published announcement in 
their newsletters, bulletins, and journals. About 65 orograms were nomina- 
ted by people whc read this advertisement or heard about the studv in snnie 
other way. 

Overall results o f the program nomination proce dure. A total of approxi- 
mately 5600 PNFs were mailed to the above search sources. From the completed 
PNFs which were returned, 963 candidate programs were identified. In addition, 
the lists and materials obtained from the National Riaht to Read Program 
yielded leads to 320 programs, and the national advertising campaign brought 
in another 65 nominations. In summary, 1348 candidate programs were identified 
through the search strategies described above. Many more programs were added 
to the pool of prospective candidates as a result of the review of literature 
described below. 

Literature Searches 

The literature review consisted mainly of two computerized searches 
and a review of major previous searches for successful programs. Descrio- 
tions of literature sources and the methods used to search them are detailed 
below. 

ERIC search . One computer search focused on the data base of education- 
al research reports cat?.loged in monthly issues of Research in Education (RIE), 
which contain abstracts of recently completed research or research-related 
reoorts. The RIE ata base is maintained by the Educational Resources Infor- 
mation Center (ERIC) and is supported by the U.S. Office of Education. ERIC 
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is a national network of clearinghouses that acquire, abstract, index, store, 
retrieve, and disseminate educational research reports and program descrip- 
tions. To assist users in retrieving the stored information, ERIC has pub- 
lished a Thes.^urus of ERIC Descriptors , a compilation of education terms 
used to index, enter, and retrieve documents in the ERIC system. The 
Thesaurus was consulted to develop a list of descriptors for use in retriev- 
ing research «nd research-related documents that focused on reading. Print- 
outs were reqiiested in abstract format. The remainder of this section on 
the ERIC search describes the strategies employed for refining the computer 
search, for screening the abstracts provided on the printout, and for using 
microfiche co.nies of the documents to. obtain identification information 
required before mail inquiries could be sent lo candidate programs. 

To trouble-shoot the computer search strategy developed to identify 
pertinent documents, a mini-search of the January through July 1973 RIE file 
was run by the Institute of Library Research in Berkeley, California, at no 
cost to the project. Fifty-nine reading-related descriptors from the ERIC 
Thesaurus were used in the mini-search and 1053 abstracts were retrieved. 
Two AIR staff members then independently screened a random sample of 200 
abstracts from the 1053 listed. Criteria for screening each abstract in 
this sample were that instruction in reading or language arts was reported, 
and that evaluation was attempted. Abstracts were sorted into categories 
of "Good," "Questionable," and "Bad." Rater cross-classifications showed 
approximately 75% agreement. Abstracts with conflicting "Good" and "Bad" 
ratings were reexamined to increase rater agreement. 

In addition to being used as a training device for the raters, the 
mini-search printout was analyzed to edit the descriptor list prior to 
submitting it for the major searches. The analysis indicated that use of a 
single set of descriptors to identify reading programs retrieved an exces- 
sive number of irrelevant documents, and suggested that crossing these terms 
with a second set of descriptors composed of evaluation terms would have 
retrieved a more relevant set of documents for further study. It was also 
determined thai by crossing the two sets of terms, the retrieval set in the 
full-scale computer search of RIE abstracts could be reduced from approxi- 
mately 10,000 documents to 3500— a large but more pertinent set of documents. 
It was felt that this limited set would provide a higher percentage of 
solid leads to outstanding reading programs with evidence to support claims 
of effectiveness, a result consistent with Right to Read's aims for the 
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study. Of the 6500 additional documents which would not be retrieved because 
they did not carry evaluation terms, the most successful reading programs in 
this group were likely to be nominated through one of the several other search 
strategies employed in this study. 

As a result of the above analysis, three additional descriptors were 
added to the original list of reading-related descriptors, and a list of 
evaluation terms was prepared. The reading-related descriptors and the 
evaluation descriptors with which they were crossed to reduce the retriev- 
al set are shown in Table 1. These two lists were submitted to the Lock- 
heed Information Retrieval Service with a request for a search of the RIE 
data base (1968 on). The printout from this search contained 3305 
abstracts . 

Three AIR staff members trained in the abstract review process scanned 
this printout to identify and eliminate obvious irrelevant entries, to note 
entries which appeared to be leads to candidate programs, and to identify 
questionable entries for rescreening. As in the mini-search, the three 
classifications were "Good," "Questionable," and "Bad." To be classified 
as "Good," the abstract had to confirm that the focus was on reading instruc- 
tion and evaluation, and that the program operated in the United States or 
its territories. After all the abstracts were read and classified, an inter- 
rater reliability check was made. Agreement was better than 80' , higher by 
more than 5Z than the inter-rater reliability found in the mini-search. 
Re-screening of the "Questionable" abstracts resulted in their classification 
as eUher "Good" or "Bad." 

From the 3305 ERIC abstracts examined in this way, 330 (about ]0%) were 
felt to provide leads to candidate readinq programs. However, this could 
not be confirmed without examining microfiche copies of the abstracted docu- 
ments. To do this, microfiche for all the "Good" ERIC abstracts were located 
at the Stanford University ERIC Clearinghouse and read to obtain program 
identification information necessary for mailing follow-up inquiries. The 
following information was obtained, if available, from each microfiche and 
copied on a card: program title, program director, program address, direc- 
tor's telephone number, school district, school district address and tele- 
phone, target population, and date of document or the year of program opera- 
tion described in the document. The cards were then used to determine programs 
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TABLE 1 

Descriptors Used in the ERIC Search 



Reading-Related Descriptors; 



Achievement Gains 


Migrant Adult Education 


Adult Basic Education 


Migrant Child Education 


Adult Development 


Migrant Education 


Adult Education 


Phonics 


Adult Education Programs 


Prereading Experience 


Adult Learning 


Preschool Clinics 


Adult Literacy 


Preschool Education 


Adult Reading Programs 


Preschool Evaluation 


After School Centers 


Preschool Programs 


After School Education 


Reading 


After School Programs 


Reading Ability 


Basic Reading 


Reading Achievement 


Beginning Reading 


Reading Centers 


Community Education 


Readinq Clinics 


Correctional Education 


Reading Comprehension 


Corrective Reading 


Reading Development 


Curriculum Evaluation 


Reading Improvement 


Developmental Readinq 


Reading Instruction 


Directed Reading 


Reading Programs 


Early Childhood Education 


Reading Readiness 


Early Reading 


Readinq Skills 


Functional Illiteracy 


Remedial Instruction 


Functional Reading 


Remedial Programs 


Individual ized Reading 


Remedial Reading 


Inplant Programs 


Remedial Readinq Clinics 


Labor Education 


Remedial Readinq Programs 


Language Abil ity 


Rural Education 


Language Arts 


Study Centers 


Language Experience Approach 


Ungraded Curriculum 


Language Skills 


Ungraded Elementary Programs 


Literacy Education 


Ungraded Primary Programs 



(Continued) 
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TABLE 1 (Continued) 



Evaluation-Related Descriptors: 
Achievement 

Analysis of Covariance 
Analysis of Variance 
Classroom Research 
Curriculum Research 
Educational Experiments 
Evaluation 

Exceptional Child Research 
Experimental Groups 



Measurement 



Multiple Regression Analysis 
Profile Evaluation 
Program Effectiveness 
Program Evaluation 
Reading Research 
Research 

Research Methodology 
Research Projects 



to which follow-up inquiries would be mailed. This involved a further selec- 
tion process to eliminate the following types of leads: (1) program leads 
contained in abstracts of documents published before 1968 or which described 
a project operating prior to 1968; (2) program leads for which documents did • 
not supply enough address information to permit follow-up; (3) program leads 
for which address information indicated the program operated outside the 
United States; and (4) program leads which duplicated those found in the 
search of previous studies described later in this section. The results 
of this screening yielded 136 candidate programs. This figure was modified 
when checked for duplication against program nomination sources. Follow-up 
consisted of mailing Program Information Forms directly to the programs 
identified, unless a document dealt with the evaluation of two or more pro- 
grams. In these cases, the evaluator or author was mailed a Program 
Nomination Form so that he could specify the project site and the appro- 
priate person to whom a Program Information Form should be mailed. 

SSIE search . A second computer search was designed to identify reading 
research projects in the Smithsonian Science Information Exchange (SSIE) data 
base of basic and applied research in life, physical, social, behavioral, and 
engineering sciences. The addition of the SSIE data base to the computerized 
literature search was necessary in order to canvass reading research projects 
currently in urogtcss and funded by both public and private sources of support. 



- 15 - 



25 



The ERIC data base was estimated to be deficient in both these respects. The 
same two lists of descriptors thafwere used in the ERIC search were submitted 
to the Smithsonian Science Information Exchange to adapt for use in searching 
their files. The printout supplied by SSIE contained 345 abstracts for cur- 
rent projects. The three AIR staff members trained in the abstract review 
procedures reviewed and classified these abstracts, according to the same 
criteria specified above for the ERIC abstract review, as either "Good," 
"Questionable," or "Bad." An inter-ratcr reliability check was performed 
and further screening of abstracts in the "Questionable" category was carried 
out as already described in the discussion of the ERIC p^«^tract review process, 
When the additional four-step screening process described above was applied to 
the "Good" abstracts, 49 were believed to provide leads to candidate reading 
programs. This figure was modified when leads were checked for duplication 
against program nomination sources. 

ERIC/RCS lis t. In addition to the computerized ERIC and SSIE searches, 
the ERIC Clearinghour,£ on Reading and Communication Skills (ERIC/RCS) was 
contacted to obtain any further leads which might not have been covered. 
ERIC/RCS provided a list of index numbers of all documents in the ERIC system 
pertaining to reading, and this list was checked for duplication against the 
index numbers of each abstract in the computer printout. Index numbers for 
abstracts appearing in the RIE before 1968 were ignored. Using the same 
criteria as before, the 1468 abstracts for all non-duplicate numbers were 
then reviewed in back issues of RIE. From this review, 65 candidate programs 
were identified. However, this number was reduced by two-thirds when those 
leads were checked for duplication against other search sources, and when 
checked for adequate address information. 

Review of previous searches for exemplary programs . Reports of previous 
research studies which had searched for successful programs were reviewed in 
ordf.r to obtain leads to promising candidates for the study. This set of 
documents came to the attention of project staff in a variety of ways, 
e.g., through staff involvement in much of this earlier work or knowledge 
of research carried out by others. In the course of reading abstracts on the 
ERIC and SSIE printouts, additional sources of previous searches were located. 
To identify a previous search on the basis of information contained in the 
abstract, the following criteria were applied: (1) Was there any indication 



er|c 26 " " 



that a survey was made? (2) Were two or more projects described? (3) Were 
some selection criteria applied? In qeneral , if these three questions could 
be answered aff iniiati vely brised on information contained in the abstract, the 
document was classified a;; ^ previous search. Reports listing aTJ Title i .<r 
ESEA programs in a oarticular geographical area (usually a city) were not 
considered, based on the third criterion above. Several previous searches 
which were located had already been identified, for example, the severnl AIR 
catalogs of exemplary programs in compensatory education. 

For this part of the literature review, snecial procedures were developed 
to screen the reports of previous searches. When examining each uf the several 
project summaries contained in one of these docL:;;^"r,ts , an internal form was 
used. On this form, the reviewer recorded each proaram's status with resnf^cl 
to the initial screening criteria. Uniform procedures were devised for deci- 
ding various questions of program eligibility under the criteria. The!^c cri- 
teria and procedural guidelines were as follows: 

• Location--The program operates in the United States. As expected, 
almost an oroqrams covered in these previous searches met this 
cri terion . 

• Recency and Longevi ty- -Tho program has operated for at lea^.t 

1 year and should operate for 2 more years. Since it could not be 
determined whether a program was presently operating from these re- 
ports of past searches, only those pronrams which had ceased or were 
scheduled to cease operating at the time the renort was written could 
be eliminated in this screening. For the remaining programs, this 
criterion was considered "passed" pendina subseguent contact with 
the program. 

• Population and Si2e--Any population was eligible, but several pro- 
grams did not pass the "size" criterion, i.e.. they were not desinned 
for all children at a grade level or cluster of grades. These pro- 
grams were preschool or day-care programs which also failed to pass 
the "focus" criterion, below. 

• Focus of Treatment- -The program aims to increase connitive achieve- 
ment in readinn-related skills. There were three tynes of nuestion- 
able programs. The first type was the preschool pronram v;hich was 
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concerned with "school readiness" and might include language arts 
but did not stress reading readiness activities specifically. In 
this case, it was decided to contact any program which used either 
a reading readiness test or IQ test to evaluate school readiness. 

The second questionable type was the primary grades program which 
covered the whole curriculum, including reading, but did not partic- 
ularly stress a special approach to reading. It was decided to con- 
tact any programs which evaluated student achievement in reading. 
If there was definitely no evaluation on reading, no further contact 
was made. 

The third questionable type was the intermediate or secondary program 
designed to raise school achievement in general, including reading. 
It was decided to contact these programs if they included special 
reading instruction, especially if reading evaluation was mentioned. 
If a program was so i integrated that it was difficult to extract a 
coherent unit dealing only with reading, in the case of either 
instruction or testing, no further contact was made. 

• Evaluation— The procedure followed was consistent with that used 
to screen ERIC and SSiE abstracts. If a program evaluation was 
described, aside from anecdotal or opinion measures, the program 
was contacted. If no evaluation was mentioned, the contact was 
not made. However, if the report indicated that hard evidence 
of program effectiveness had been a precondition for inclusion in 
the study, this criterion was waived. 

The internal form used for initial screening of programs reported in the 
previous searches also provides space for noting questionable aspects of the 
program that would clarify the reasons for screening decisions. For every 
program which passed the initial screening criteria, the name and address of 
the person to be contacted for further information on the program were recor- 
ded in a box on the form. These persons were later sent the Program Infor- 
mation Form. **— 
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The previous searches which were reviewed, and the results of initial 
screening of programs they described, are contained in Table 2. Although 
125 candidate programs were identified in screening these reports, almost 
half of the programs were duplicated by leads discovered in other search 
sources. Ihey were added to the lists of programs to receive Program Infor- 
mation Forms. 

Review of current studies identif ying successful programs . In addition 
to the published reports of previous studies which had identified successful 
programs, two current projects which were charged with evaluating educational 
programs were also reviewed. These projects were contacted in person or by 
pfione in order to discover whether any of the projects studied met the ini- 
tial screening criteria of this study. The results of these contacts are 
summarized below. 

1 , Cooperative Longitudinal Study of Demonstration Education Programs. 
This study was being conducted by AIR to identify the effects of 
educational approaches in various contexts. Discussion with the 
director of this project revealed that the data base was not designed 
to permit evaluation of any particular project, but rathe of dif- 
ferent elements of educational treatments and their effect on students 
with various characteristics. No leads to successful projects could 
therefore be obtained from this study. 

2. The Development of Project Information Packages for Effective 
Approaches in Compensatory Education . Meetinns were held with the 
contractor for this study to insure that projects would not be 
duplicated, and to obtain any other relevant and helpful findinns 
from their efforts. No new leads to successful projects that were 
not to be included in that concurrent effort could be obtained, 
based on a review of project reports and materials supplied to AIR 
by the contractor. 

Overal l results of th e l iterature searches. Well over 3500 abstracts 
and about 350 project documents were screened during the search for candi- 
date programs. The computerized literature search of the ERIC and SSIE 
data bases yielded about 185 program leads. The list of abstracts provided 
by ERIC/RCS contained 65 leads. In addition, the review of nearly a dozen 
major previous searches for successful programs yielded another 125 program 



- 19 - 

29 









■o 








0) 








E 












o> 




o 








CL 








o 




^- 




"O 




0) 


UJ 


c 




0) 


CO 


0) 


< 






u 




(/) 




(/) 




















-P 




(/> 




(/) 




3 




O 








> 












D. 



-p 
c 

0) 



o 



0) 



0) 



13 "O 
Q- 3 
-P 

E 

<t3 0) 

u x: 
o 

Q- O 

^- 
^- ^ 

o T- 

P 

3 r- 

O 0) 



0) 

u 
o 




0) 



p 
to 
c 



0) 



0^ <T3 <0 
r- -r- 

f» (X) 

P P c 

U O 

O. U *P Ll. 



0) 



<t3 
O U 

C 

U) r- 
•I- C Q- 
x: 0) 3 
P 0) "O 

c u u 

•I- U) O 
N -P 0) 

•I- ^ 

<d 0) 0) 
E u) -c 

E CO U 
CO O. 0) 

w 

CO "O 0) 
-P c 5 
u <0 
3 CO 
"O "O "O 
O 0) <d 
W P 0) 
Q- <0 r- 

r- W l/> 
r- I 0) 
C7>-C 
0) C P 

-P "O 0) 
fO u 
0) C 

o w o 



p 

x: P 
0) p 

U 0) 

0) o u 

2 CO 0) 

x: 

u) u) 2 
E V. 
C 0) U) 

o x: 0) 

CO -P 



C r- 
O ^ 
•I- 3 
P O. 

C U 
•I- o 
E 

O l/> 
0) 

E Q- 
<0 O 
W r- 
C7> 0) 
O > 
^ 0) 

-P 
l/> (J 
0) 3 
U "O 
W O 
3 W 
O Q- 

0) 

^ x: 
u -p 

<Q O 
0) *P 

"O 
^ 0) 
0) r- 

x: 

-p <o 
o E 



o 

CO 3 

U to 

•I- U) 
if. 0) 
•I- U 

u u 

0) 3 
O. (/) 

"O 

<Q <U 
I/) 
0) 3 
-P 

c c 
E 0) 

o x> 
c 

3 

O -P 
U U 
3 

0) o 
x: w 
*p o. 















-p 1 


« 














x: rd 












c 




o> u 


.p 










o 




•1- 3 


u 
















3 


0) 




a> 










TD 


■o 




■o 










O 


o 




o 












z: 




z: 




i 




o 


cx 














• -p 


-p 


• 


• 


• 










c 




c 


x: 








<d 


0) 


u 


o 


u 




x: 




• Q. 


E 










u 




Q E 


cx 


<t3 


p 


<t3 










o 


0) 


<t3 


0) 




rd 








l/> 


U 


U> 




0) 




r- 0) 




0) 


3 


0) 




tn 




•1- x: 


> 




■o 


QC 




0) 




> -P 


0) 




0) 






o: 




x: 


"O 














o ^- 




O "O 


O 








o o 


"O 


^- 


o 




• 


fd 




p 


c 




o 






c 




rd c 


rd 




x: 


U> 


C 


o 




U O 




0) 




0) 








^ 


x: 


p 




P "O 


p 




P 


u 


3 




3 




rd 




#^ rd 




p x: 


P 


0) 


U 




• 3 


rd 




u 






3 




r- 


0) 


p 




P 




■o 




rd 


l/> 


i/> 








LU 




• > 


0) 


c 




C 














E 






rd 










fd 








• 






c 




c 


£ 


r 






rd 


fd 


CT 


rd 






■o 


U UJ 


c 


u 


O 


U 


o 


o 


3 


o 


o 










^- 


P 


^- • 






Q. 




a 




CO 




p 


0) 




0) 








rd 




Am 




Am 




Ca 




Cr 





p 
o 

0) 



0) 

p 
c 



U) U 
0) o 

x: "o 

p 

0) 

c 

•I- x: 
c *p 

0) 

0) c 
u 

iA "O 
0) 
C N 



c 

<d 

fd 
0) 

p 



x: 
u 
s. 
fd 
0) 

CO 



u x: 
-p 
o> o 
c 



fd 0) 
E 0) 



> 

•I- 3 U 

fd u) (/) 

u) -P fd 
fd u 

5 0) -P 

c o c 

O W 
•I- Q- 

W 0) 

0) LD x: 

-P LD -P 

u x: 0) 

•P CO 

C (/) 

o ^- fd 

•I- O Q- 
-p 

o 

• ro 
I/) 
-p 

U (0 

0) *P 

O 0) 

W E 
CL 3 



fd 

3 

fd 
> 
0) 

0) 

x: 



3 

o 
^- 

i/) 

fd 
0) 



•a 

0) 

-p 
fd 
u 



o. 

3 
"O 

-P 

o 
c 

0) 

w 



3 



P 

c 
o 
o 



0) 

a 

3 
O 



CO 



c: 
fd 



fd 



20 



ERIC 



30 



C 



C 
O 

o 



UJ 

-J 

CO 

< 



c c 














o 


•p- 
























c »— 






















c 






O 




«^ 










-C {A 














10 to 


> 


E 




TD 






a; 


o 






<D 






Q) u 












u 




a> 




'p- 






Q) :3 




o 










5 on 




pr 




Q r 


ed 




SZ (1) 


O 








LO 




u ^ ^- 


a; 






LO 




•r- 




LO 






fT3 




-C O 


a; 


a; 










Q--C 


• 










-P 






CO 




LO fT3 


o 




•p- 


•P 






E -M 


-p 






CO 










o 


a; 




u 






c 




-p 


a; 


• p- 






0) 




•p- 


> 


-C 




o c 




• 




o 






c -o 


LO 


u 








Q..P- 














-P 


*p- 


0) 






o 




O fT3 




> 


C 


(D 








u 


0) 


• p- 


-C 












c 


-P 












0) 




-p 




q; > 


0) 








u 










O 


a; 


• 




fT3 


-P 


u 




*o 




•1- 


-P 


U 


LO 


c 


o 


•p- 




C 


<D 




o 






C fT3 




CL 


^ 


*p- 


o. 


(D 


0) -C 


> 


X 




-p 




-P 








•p- 






•p- 




fT3 




-P 








O 


LO 




•p- 


C 




u 


>> 


•p- 


O 




• p- 






"O to 






• p- 


-P 




a> 






LO 




C 


C 


c 


+J CO 


C7^ 




0) 


o 


o 


•p- 


10 fT3 


C 


O 




u 


•p- 


c 




•p- 


•p- 


-p 




-p 


<D 


10 


o. 


> 




LO 


*p- 


<D 










•p- 












0) 


-C 




u 






o. 


E 


h- 


fT3 


LO 



LO 
-P 

O fT3 



0) 
-P 

c 

> 



fT3 



oes E 

<D 

. 0) 
CO 

#» <D 

>^'— 
^ 0) 
LO LO 
O. 

f— O 

x: >^ 

^ -P 
• LO 



LO 



Q 

• LO 

''o: E 

T3 O cr 

•p- o 

s- * s- 



T3 
Oi 

^ U 
• LO 



•p- 

Q 
r o 

• 3 

a. -P 

LO 

r 

3 < 

fT3 

0) 

Q. • 

E i-' 
to 

O • 

CL 



-P 
-P 
. (U 

Cj ^ 
U 

.r- 
S- C 

^ (n 



LO 

E 
S- 

CTi^p- 
o 
s- 

Cl. 



U 



-P 
C 

> 

fT3 



I 

O 

c 

O »— Q. 
'p- (T3 

•P C O 
(T3 O -P 

t -P >, 
O U 

^- u o 

•p- T3 U 

UJ o; 

q: 'p- 
uj o -a 
—J ^ 

>> 

0) LO 

o 

-P -P 
o o 

-P fT3 . 
U — I -P 
3 C 

-P q; 

o LO E 

a; o. 

Q-3 o 

fT3 0) 

fT3 > 

LO U. O) 

.p- Q 
O) 

O -P C 

»— fT3 

-P O -C 
fT3 U 
U E 

LO -P O) 
•p- LO LO 
-C >> CU 

h- LO q: 



<D 'p- 

-Q 

c c 

•r- fT3 

O) LO 
-P 
V. 

C 

•p- q; 

f— 3 

U C7> 

C C 

•p- (O 



0) 

E 
o 

LO LO LO 

o <D -a 

O -P 



c 
o 

•p- 

LO 

u 

c .-0 



P 0) »— 

>> :2 X) 



LO -C 
m LO 

q; .p- 

C 



in 



LO 

E 
O 

o. 



T3 
C O 
•p- O 

-C 
LO T3 
E ^ 
fT3 'p- 

^ x: 

C7> U 



LO fT3 
LO 'P- 

U -P 
U 

LO I— 
<D 

fT3 O 

O LO O 
Q-'p- LO 
O »— LO 

P U rT3 
J2 

C +J 
(D Oj 
> O -C 
p- Q--P 
C7> CI, 

3 C 
LO LO O 

5 o -o 

O C 3 
O »— 
fT3 'P- U 
— -P -P C 
'p- 

3 U 3 

c 



LO I— 



o 

'p- 

o. 



LO 

E 
to 

a> 
o 



O rT3 

c x: 

-C E 
>> 

-p 

o. 



CVJ 

c 

> o 
a; 



to 

3 



•r- C 
f— "p- 

o 
a; LO 

LO 

CO to 
f— X) 

q; 

>>-c: 
-P c 

LO O 

a> c 
o 



<D to 

x: > 

4-) O) 



(D 

5 



LO 

to 

U 
LO 



u 

Q- > 



to 

C 

O -P 

.p- u 

-p .p- 

•p- 

to i/) 

4-1 C 

O O) 

4-> JD 



O 

P C7> 



C LO 

C > 

'p- C 

O. O) 

CO 4-> 

m 

c o 



T3 
3 



C 
O 

CJ 



C7> 

c 

•p- 

c 

u 

LO 



o 
o 

u 

o 

LO 



LO 
4J 
U 
d) 

o 
o. 

c 

LO 



fT3 

cn! 
o 

h- Q-1 

a: 

UJ 

-J »— 
< 3 
U 

•'"^ 

UJ 3 
U 

z s- 

• 4J 

00 C 
(D 

^ E 
(D (D 



C 



(D 



21 



31 



0) 



o 
o 



c -o 

S to 
> 

O <t3 

Q--0 
E CU 
1-4 ^ 

U 

c -o 

•I- C 

-o O 

q: ^ 
u 

E 

o 0) 

-O r- 
0) O 
C O 

^ s. 

O 0) 
-J 

S. 

o a 

(U c 



I/) 



CU 



I/) 

CU (/I 
C7> CU T5 
(QUO) 

4-» a ^ 

(tJ l/> I- 
u 

to I/) 



0? 



O • 

I/) 

I/) 

r- CU 

3 

to 

CU 

a: 00 
• <u 

D -M 
C 00 
•I- 

s. 

c 

-o 
to a; 
E c: 

o c 
o 

Q- U 

CU CU 

3 



rO ft3 CU 
I/) "O 

-O O CU 

00 S. 

=> CU 

O i 

^ CSJ 

I/) 

E "O 

<t3 CU ^ 

L. CU 

O . <t3 I/) 

O C CU 

a)x: 
I/) 

O) CU 

c -o o 
-o c 

<Q CU • 
CU CU -o 
S. ^ CU 

r- "O U 
(Q 

•I- -c 

-o c: 
CU o 
E U U 



CU 
CU 

i- I/) 

-O 1— CU 
3 T3 



< O O to to U 



r- U 

3 0) 
X) i/) 

I. 

CSJ CU 
r- o 

^•*= 

CO 3 
3 O 

c s. 

-O 

o o; 

I/) M- 

cn c 
c <u 
•I- -a 

L. C 
CU CU 
Q. CU 
O ^ 

•I- (O 

+-> a; 

00 s- 



o 

CU 

-o c 
X) o 

I/) <t3 
<t3 E 
S t 

o 

<t3 C 
CU 

E 

CU ^ 
C C7> 
O 

CU 1. 
C CL 

o 

CU 
CU ^ 



1. 
o 

3 to 
t/) 1— 
CU CT 



O fO 
M- E 



0) 

^ t 
•M CL 

1— S. 
C CU 
O -C CU 

" sz 

C C -P 

O CU 
-C r- > 
to 3 O 

CU O <t3 
00 

CU c; E 

5 u C 

u o 

-C 3 M- 

U 00 S. 
•I- CU 

^ Q) CL 

O 

to O •*-> 

U C 
CU C CU 

O I. 
O -O 

-M r- 

3 -C 

o 1— a 
> -o 

-O CU CU 
C 

3 -O ftJ 

O C 4-> 

3 C 

O rO 

to > 

-o -o 

3 M- fO 
•P O to 
I/) 'f- 

i/) "O 
I/) 'f- 

I/) 

x: ro C 
^ 



CO 



to 
E 
rt3 

a> 
o 
1. 

Q. 

CU • 
I/) <t3 
CU 

sz 

•M CU 



O S- 
U 

• O) 
uo c 

> CU 
<U CU 
r- I. 

u 

-O I/) 
CU 

-*-> r- 

a <t3 

CU 

CU c 

O CU 

to -*-> 
3 

o -o 

•I- CU 

> to 

Q) to 
U fO 
Q. Q. 



CD 
< 



3 
-*-> 

CU 
CU 



O 

<u 
c 
s. 
<u 
> 







I. 




"O <o 




2 <- 


-0 




E Q. 


CU 




<— E 


01 




n: ^ 






X 


-*-> 




CU 


c 










> 




0 












• c 


I/) 




~J 0 


















ex. ro 






c 


C 










3 E 












CU X 


a 




Q. a) 


3 




E 


-0 






CU 


















0 




• 


^- 




0 3 






U. 




c 




E 


CU 












-0 




cn 




0 


0 




a> • 








0. 








3E 






32 



- 22 - 



leads. These numbers, especially the number of leads supplied by the review 
of the ERIC/RCS and previous search materials, were substantially reduced 
when checked for duplication against other search sources, and when checked 
for adequate address Information. 

C ombined Results of the Searches for Ca nd idate Programs 

The results of the program nomination and literature review strategies 
are shown in Table 3. These numbers include about 200 duplicate leads. In 
round figures, over 1700 leads to candidate programs resulted from AIR's 
systematic, nationwide search. About S0% of these leads were supplied by 
the program nomination procedure, and 20% by the literature search. The 
most productive nomination sources were professional organizations and 
groups, and federal sources--particul arly f-ie National Right to Read Proqr,i„i-- 

which supplied lists of programs. The best sources of leads to candidate 
programs in the literature search were the ERIC data base and reports of 
previous searches for successful programs. 

The most productive source of leads to candidate nronrams, therefore, 
was the program nomination procedure. However, the literature search sun- 
plied over one-fifth of the candidate programs and so may be judged to have 
been a worthwhile activity. The mu.t economical source of leads was provided 
by Right to Read and other federal sources in the form of lists of school - 
based and community-based reading programs. 
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TABLE 3 

Results of the Search for Candidate Reading Programs 
by Major Search Source 



Source 



Number 
of Leads 



Percentage 
of Total 



Program Nomination Sources: 

Program Nomination Panel 
Federal Sources 
State Sources 
Local Sources 

Professional Organizations 
and Other Groups 

Response to Advertisement 
Total Leads from Nominations 

Literature Search: 

ERIC Search 
SSIE Search 
ERIC/RCS 

Reports of Previous Searches 
Total Leads from Literature 



46 

349 
171 
128 

589 

65_ 

1348 



136 
49 
65 

125 

375 



3 1o 
20 
10 

7 

34 
4 

78 % 



8 1 
3 
4 
7 

22 1 



TOTAL LEADS FROM ALL SOURCES 



1723 



100 % 



Note. The numbers in this table include about 200 duplicate leads. 
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Chapter III 



THE CRITERIA FOR ASSESSING PROGRAM EFFECTIVENESS 
AND THE PROGRAM INFORMATION FORM 



Criteria for Asse ssjn^ Pr ograni Effectiveness 

The screen and review process was that of reordering the large quantity 
of raw program information on the basis of criteria for judging program 
effectiveness. Screening criteria were the key organizing concepts opera- 
tional ized through data collection instruments and procedures that would 
permit comparisons on common, objectively measurable standards. The major 
instrument developed to assess program effectiveness was the Program Infor- 
mation Form (PIF) which contained precoded items enabling all programs 
completing the PIF to be objectively scored. The PIF and *ts accompanying 
cover letter are shown in Appendix D. 

Information from the PIF, together with available program description 
and evaluation documents obtained from the local programs, served as the 
primary data sources for programs selected for intensive review by AIR 
staff. All of these sources, as well as direct telephone consultation with 
program staff, supplied the information used to prepare description and 
evaluation summaries for recommended exemplary programs submitted to the 
Dissemination Review Panel (DRP) of t!'^: Office of Education. 

The stipulations of the National Right to Read Program regarding the 
criteria for assessing program effectiveness were clear; these provided a 
firm skeleton onto which it was relatively easy to put flesh. The criteria 
are listed below, with appropriate PIF items noted in parentheses: 

• Location --The rrading pro'.jrani must have been located within the 
United States, its territories or possessions (Items 4, 5). 

• Longev ity--The program must have beer, in continuous operation for 
at least 1 year and must also have had plans for continuing opera- 
tion for at least 2 more years (Items 7, 8). 

• Av ailable and R..cent Docu ment ation --Program description and eval- 
uation reports must have been available and based on evidence 
obtained since 1 January 1968 (Items 9, 10, 11, 20). 

- 25 - 
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• Fpcus^- -Evaluation data must have been focused on cognitive gain 
and academic achievement (Item 6). 

• Measures --The measures of program effectiveness must have been 
reliable and valid, preferably obtained with nationally standar- 
dized reading tests, although locally developed instruments with 
adequate measurement characteristics were acceptable (Items 13-19, 
58, 60). 

• Comparisons --A sound basis for comparing the effect of the reading 
program upon its participants must have been established. Comparisons 
of pretest and posttest score gains for program participants with 

an appropriate control group were preferable, although comparisons 
with a norm group or with the achievement of pre-program groups 
were also acceptable if these were persuasive and compelling (Items 
21-26, 53). 

• Sam£ljna- -Generalization from the participant sample to a defined 
target population should have been evident. In particular, the 
effects of attrition upon the soundness of the interpretation of 
positive program impact should have been minimal (Items 29-43, 59, 
62). 

• Size--The evidence of positive program effect must have been 
based on samples of sufficient size, on more than one classroom, 
and over more than a single year preferably, so that explaining 
resiiits in terms of a single teacher or group of students was not 
ft credible rival hypothesis (Items 11, 12). 

• $ignificance--ln assessing program effect, appropriate statistical 
tests and evaluation designs must have been used in summarizing the 
evidence. One-tailed tests must have lead to rejection of the null 
hypothesis at the .05 alpha level. Furthermore, educational as well 
as statistical significance must have been observed so that trivial 
positive differences resulting from extremely large sample sizes did 
not lead to claims of statistical significance only (Items 27, 44, 
45, 47-60). 

• Exportability --Instruct-;onal materials, activities, services, 
organizational details, and key procedural information must not 
have precluded the possibility of widespread dissemination (Items 
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28, 46, 63-69, plus two open-ended items requesting detailed 
staffing, objectives, and assessment information). 



The P rogra m Information Form {Plf) 

To spare respondents wasteful effort, items relating to the four 
crucial criteria were placed at the beginning of the PIF and the least 
satisfactory answer for each was marked with asterisks. Respondents who 
had selected one or more of these marked answers were asked to complete 
and return only the first three pages of the PIF (which included identify- 
ing information). 

Each of the requirements imbedded in the criteria above were measured 
by items found within the PIF, Some items could be answered "Yes" or "No," 
for example: 

Item 6: Is cognitive improvement in reading and 
reading-related skills a major focus of 
you^^ program? 

YES NO 
Answer: PJ (6) 

For others, a range of answers was provided. In these cases, the response 
range commenced below the threshold for selection and continued beyond it. 
This was done both to avoid, straining the honesty of the respondents and to 
permit room for flexibility in assessment. For example: 

Item 9: Are evaluation reports (e.g., baseline 
test data, retest data, measures of the 
program's effect) available? 



Answer: None available and none 
pi anned 

None available but initial 
steps taken 



j ** (9) 

□ 



Available but not published 

Most recent publication 
prior to 1/1/68 



□ 

Most recent publication P j 

since 1/1/68 
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In some cases, it was necessary to break the question into a number of 
parts, with "Yes-No" answers to each, for example: 



The Program Information Form consisted of three sections. Section I, 
"Identification Information," contained five questions to identify the 
program and key personnel. Section II, "Program Screening Information," 
was designed to permit immediate coding for subsequent ca^-d punching and 
computer analysis. Section III, "Brief Descriptive Information," was 
designed to obtain certain program information that was not amenable to a 
precoded format. 

Scoring PIF items . There were 55 PIF items, starting with Question 6, 
on which programs were scored. These 55 fell into six categories; 

• Criteria (i.e., conditions imposed by the RFP) 

• Statistical Adequacy 

• Experimental Design 

• Comparison Claims 

• Other Considerations 

• Target Populations 

For each category except the last, scores were derived by allocating varying 
scores for answers with varying degrees of acceptability. Information on 
scoring PIF responses is summarized in Tables 4, 5, 6, 7, 8, and 9. 

Response to the PIF mailing . Searches of the literature, together with 
responses to the Program Nomination Forms, produced a list of 1520 programs 
to whom PIFs were sent for completion. Of these, 804 PIFs were returned to 
AIR for data processing. Nearly 100 of these were so incomplete that they 
were unusable. 



Items 54-57 



Which of these factors were taken 
into explicit account in the analyses 
of BOTH program AND comparison data? 



YES NO 



Answers: 




Ethnic propor- 
tions in group 
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TABLE 4 

Category 1 : Criteria 
(5 questions; maximum score 9) 



QUESTION # 



6 ** 



** 



10 



** 



12 ** 



27 



QUESTIONS 



"Is cognitive improvement in 
reading and reading-related 
skills a major focus of your 
program?" 

"Are evaluation reports (e.g., 
baseline test data, retestdata, 
measures of the program's ef- 
fect) available?" 

"Evaluation data are available 
for how long?" 



"How many participants or indi- 
vidual records are included in 
the evaluation?" 



"How significant were the sta- 
tistical results showing the 
effect of your program?" 



COMMENTS 



Answer "Yes" essential . 



Five choices, but availa- 
bility essential; extra 
point for "since 1/1/68." 

Six choices, but availa- 
bility essential; extra 
point for "more than two 
years. " 

Seven choices, but more 
than 10 essential ; one 
point for "more than 30" 
and extra point for "more 
than 50." 

Five choices; one point 
for "better than 5 per- 
cent one-tailed" and 
extra point for "better 
Lhan one oercent." 



** Respondents who marked one or more of these questions with 
unsatisfactory answers were asked to complete and return 
only the first three pages of the questionnaire. 



ERIC 



-29- 39 



TABLE 5 



Category 2: Statistical Adequacy 
(5 questions; uiaximuni score 8) 



QUESTION # 


QUESTIONS 


COMMENTS 


7 


"How long has your program been 
operating continuously?" 


Five choices; one point 
for "at least one year" 
and extra point for 
longer. 


11 


"Are the data evaluating your 
total program approach available 
for one or more sites?" 


Three choices; one point 
for "your site only" and 
extra point for "more 
than one site." 


46 


"In this program, how many hours 
per week are scheduled for the 
subject Language Arts?" 


Seven choices; one point 
if less than six hours. 


53 


"How large was the estimated 
program effect on achievement 
(i.e., the average gain of stu- 
dents in the program over and 
above the gain expected in a 
comparison group)?" 


Seven choices; one point 
if "one-quarter of a 
standard deviation unit" 
and extra point if larger. 


58 


"What was the reliability co- 
efficient of the test used to 
measure reading achievement 
for this program?" 


Seven choices; one point 
if 0.7 or larger. 
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TABLE 6 



Category 3: Experimental Design 
(21 questions; maximum score 18) 



QUESTION ff 


QUESTIONS 


COMMENTS 


13-19 


"What measures have been ana- 
lyzed to show the success of 
your program?" 


Seven "Yes-No" options; one 
point for (14) , Analysis of 
locally developed reading 
test results; also, two 
points for (13) , Analysis 
of nationally standardized 
reading test results. 


44, 45 


"Are specific diagnostic tech- 
niques or instruments used to: 
(44) DetermiTie -each studetit's 
level of reading readiness or 
skill (e.g., his reading grade 
level)? (45) Determine each 
student's strengths, weaknesses, 
and difficulties in language 
and reading skills (e.g. , 
difficulty with decoding)?" 


One point for each "Yes." 


^7-52 


"What summary statistics were 
used in the analysis of pro- 
gram data?" 


Six "Yes-No" options. One 
point for each "Yes" to: 
(47) Means or medians, (48) 
Standard deviations or var- 
iances, (49) Covariances 
or correlation coefficients, 

(50) Frequency counts, per- 
centages, or proportions, 

(51) Significance tests. 


!>4-57 


"Which of these /actor?, were 
taken into explicit account in 
the analyses of BOTH program 
AND comparison data?" 


Five "Yes-No" options . 
One point for each "Yes" 
to: (54) Age, (55) Sex, 
(56) Grade Level , (57) 
Ethnic proportions in 
group . 


59 


"What percentage of annual at- 
trition or loss of students 
from the program was allowed 
for to correct for bias in 
statistical analysis (e.g., by 
eliminating from consideration 
persons who start the program 
but do not finish?" 


Six cnoices; one point for 
any answer except "No al- 
lowance was made for 
losses." 


60 


"How similar were the pre- and 
posttests used to determine 
gain in reading skills?" 


Seven choices; one point for 
"Only one test has been applied"; 
two points for either "Were 
parallel forms of a single 
test" or_ "Were consecutive 
forms from the same source." 
No points for any other answer. 
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TABLE 7 

Category 4: Comparison Claims 
(6 questions; maximum score 9) 



QUESTION # 


QUESTIONS 


COMMENTS 


21-26 


"What kind of improvement or 
gain by program students was 
found?" 


Six "Yes-No" options. One point 
for each "Yes" to: (21) The mean 
test score of the students ex- 
ceeds a specified norm; (23) A 
mean gain for less than one year 
is bigger than expected. Two 
points for each of: (22) A mean 
gain over exactly one year is 
bigger than expected; (24) The 
mean of students iji the program 
exceeds that of comparable stu- 
dents not in the program. Three 
points were given for response 
(25), The mean gain of students in 
the program is greater than for 
comparable students not in the 
program. 



TABLE 8 

Category 5: Other Considerations 
(3 questions; maximum score 4) 



QUESTION # 


QUESTIONS 


COMMENTS 


8 


"Do you plan to keap your program 
operating for at least two more 
years (through the 1974-75 academic 
year)?" 


One point for "Yes." 


20 


"Are up-to-date program descriptions 
available (e.g., staff, participants, 
schedules^, and activities)?" 


One point for "Yes." 


28 


"By what amount does the annual per- 
pupil cost of this program exceed 
that of the regular district pro- 
gram?" 


Eight ranges: two points 
for "less than $100," one 
point for "between $100 
and $199." 



TABLE 9 



Category 6: Target Populations 
(13 questions, used for classification only) 



QUESTION # 


QUESTIONS 


COMMENTS 


29-35 


"For what target population of 
students is your program de- 
signed?" 


Seven "Yes-No" options, 
including ones for "Men- 
tally retarded," "Bilin- 
gual," "Disadvantaged," 
etc . 


36-41 


"Are 20 percent or more of your 
program students in any of the 
following categories?" 


Six "Yes-No" options, 
including ones for 
"Black," "Spanish-speak- 
ing," etc. 


42 


"In which area do the majority 
of the program students live?" 


Six descriptors on a rural- 
to-urban continuum. 


43 


"What is the average family in- 
come level of students in the 
program?" 


Three income ranges. 



o 
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Chapter IV 

SCREENING EFFECTIVE READING PROGRAMS 
AND REASONS FOR REJECTION 

Tj'g_,Seguence of Screenings 

There was no single operation that identified exemplary programs. 
Rather, a sequence of four successively tighter screenings was applied: 

• First, the literature search combined with the individual 
and organizational nomination procedures defined a first- 
stage screen to locate effective programs. Criteria 
related to program effectiveness, recency, focus, loca- 
tion, size, and longevity were outlined both for nomi- 
nators and for AIR literature reviewers. 

• Second, as indicated in the last chapter, responses to 
the PIF were summarized into five subscale scores related 
to indices of program effectiveness. 

• Third, the 728 programs supplying complete PIF data 
were rank-ordered on the PIF composite score. For all 
programs with PIF composite scores of 30 or higher 
(78% of the total), available documentation was re- 
viewed by senior AIR evaluators. Of these programs, 26 
programs were recommended to the Dissemination Review 
Panel (DRP) of the Office of Education as exemplary 
programs. Of other programs located and reviewed later, 
an additional program was recommended, bringing the 
total to 27. 

• Fourth, the DRP reviewed the AIR recommendations. This 
panel's decisions to approve or disapprove programs for 
dissemination were found to be consistent with the second 
and third stage evaluations carried out by AIR. DRP 
approved 14 of the 27 programs AIR recommended for 
packaging; DRP action is pending on 2 programs. 



This four-stage screening procedure, imposing successively more 
rigorous selection standards, would appear to be justifiable and fair. 
While errors of selection or rejection are possible, as in all human judqe- 
ments, the consistency of the agreement with which programs were evaluated 
at the various stages was considered to be convincing evidence that such 
errors were reduced to reasonable limits. 

The remainder of this chapter will describe the second and third stage 
evaluation procedures and the results of fourth stage screening by the DRP. 
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PIF Second Stag e Scr eeni n g 

As PIF replies were received, they were at once screened, obvious 
errors corrected, marks made to give the benefit of the doubt where called 
for, and the five categories of items described in the previous chapter 
were hand-scored. Those programs that passed all the criteria covered by 
PIF items marked with asterisks and had Experimental Design scores of 14 
or higher, were given closer scrutiny. For those programs that appeared to 
be likely prospects, th-^" PIFs were photo-copied and the accompanying docu- 
ments passed on to evaluators for immediate review. 

As a substitute for a pilot run, the first 65 cases were put through 
the computer, which revealed commonly occurring errors in coding that needed 
careful attention before card-punching. Thereafter, PIFs were accumulated 
in batches of approximately 100 for card-punching. Each PIF was assigned a 
five-digit identification number, with the fifth digit indicating additional 
reports from a single site. These code numbers occupied the first five 
columns on the cards, with coded answers starting with the sixth. Double- 
punching was avoided, and all card-punching verified. 

When 287 cards had been accumulated, they were again scored by computer, 
category scores accumulated, and the summary statistics computed for each. 
As could be expected in view of its greater length, the Experimental Design 
category had a standard deviation about twice that of each of the others 
except for Comparison Claims. These scores were doubled for the categories. 
Criteria and Statistical Adequacy, in order to give them equal weight with 
Comparison Claims and Other Considerations. The composite score for the 
whole PIF was therefore calculated as follows: 

PIF Composite Score = 2 Criteria + 2 Statistical Adequacy + 

Experimental Design + Comparison Claims 

+ Other Considerations. 

The first four categories thus carried roughly equal weights and the 
last about half as much as each of the others. This scoring system was 
maintained thereafter for all PIFs. A check on the; -nummary statistics 
after 728 PIFs had been processed showed only trivial changes from those 
of the first 287. These first 287 were scored on the composite, the print- 
out showing not only composite score, but fiU,o the individual category 
scores and the individual question scores from which the composite score 
was derived. Programs were arranger' in order of their code number, making 



ERIC 45 



- 35 



it easy to ref>,'.r to all data for a particular program, and to begin the 
next stage of review while waiting for remaining PITs to be returned. 

The return of completed PIFs had virtually ceasod, except for one or 
two a week, when 728 were received. At this point, a final printout was 
called for with all statistics; the more technical aspects are summarized 
in Tables 10, 11 , 12, and 13. 

The composite scorj and empirical results. Table 10 presents the 
frequencies of scores that resulted, with descriptive statistics. 

TABLE 10 

Distribution and Univariate Statistics of the PIF Composite Score 



Sc ores Frequencies ".. Cum. 



60+ 


1 


0. 


1 


0. 


1 




55-59 


12 


1 . 


7 


1. 


8 


Mean = .56. 4 


50-54 


51 


5, 


6 


7, 


4 


Standard Deviation =^ 9 J 


45-49 


SA 


11 . 


6 


19. 


0 


Skev/ness = -0.3 


40-44 


139 


19 


0 


38 


0 


Kurtosis = +0.3 


35-39 


133 


19. 


7 


57 


7 


(The rd\v]t of scores, 


30-34 


148 


20 


3 


78 


0 


20-f)f^ :^.hr;wed a close 


25-29 


103 


14. 


2 


92. 


2 


dpprox iniation to the 


20-24 


31 


4. 


2 


• 96. 


4 


Noniiril Dislributi oh, ) 


15-19 


16 


2. 


2 


98 


6 




10-14 


4 


0 


6 


99 


n 




5- 9 


5 


0, 


7 


99 


q 




0- 4 


1 


0. 


1 


100. 


0 




TOTAL 


728 


100. 


0', 








Statistical 


summaries of 


caJ;egor^ 


scores. From Table 11 it will be 


that Criteria scores 


had 


a 


mean 


i n 


the upper half of the range, though 



with only a small amount of negative skewing. Infrequency jf low scores 
here is due to the fact that respondents were invited to save effort and 
not to complete the questionnaire if several answers in this section were 
below tlie cut-off. Experimental Design had a relatively liijh correlation 
with the criterion, since satisfactory answers on Criteria scores were 
logically necessary to Experimental Design, so tliat here too the mean is 
fairly high witi some negative skewing. The distribution for Otiier Consid- 
erations, however, is fairly well negatively skewed, with a mean located 
at three-quarters of tfie range, meaning that most respondents obtained 
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scores of 2, 3, or 4. In fact, only 42 programs scored below this. The 
high negative kurtosis was due to the fact that all 142 cases were dis- 
tributed on the two modes, at 2 and 4. Comparison Claims had a low mean 
and relatively high standard deviation. Here too the distribution was bi- 
modal, at the scores 0-1, and at 3. 



TABLE 11 

Univariate Statistics of Category Scores (before weighting) 



Category 


Mean 


SD 


Skewness 


Kurtosis 


Range 


Criteria 


6 


.09 


1 


.45 


-0 


.21 


0 


.30 


1 


-9 


Statistical Adequacy 


4 


.10 


1 


.41 


0 


.03 


0 


.14 


0 


-8 


Experimental Design 


9 


.87 


3 


.45 


-0 


.31 


0 


.11 


0 


-18 


Comparison Claims 


3 


.11 


2 


.85 


0 


.67 


-0 


.72 


0- 


-9 


Other Considerations 


3 


.04 


1 


.02 


-0 


.43 


-1 


.27 


0- 


-4 


Composite 


36 


.39 


9 


.08 


-0 


.27 


0 


.30 


3- 


-60 



Scale intercorrelations are shown in Table 12. The most noteworthy 
fact is that Other Considerations had very little relationship to the 
other four scales. This, together with its small standard deviation (1.02) 
and only unit weight, accounts for the low correlation with the PIF com- 
posite score. In fact, even that is largely the result of the part-whole 
correlation. As intended, the Other Considerations score contributes very 
little to the PIF composite score. 



TABLE 12 
Intercorrelations of Scores 



Category 


Intercorrelation with: 
2 3 4 5 


PIF 
Composite 


Criteria 


.48 


.45 


.35 


.06 


.76 


Statistical Adequacy 




.35 


.31 


.09 


.70 


Experimental Design 






.44 


.13 


.78 


Comparison Claims 








.09 


.70 


Other Considerations 










.22 



Reliability of the PIF composite score . Correlations of the individual 
scales with the PIF composite scoru are, of course, all sharply raised by 
the part-whole effect. However, a conservative estimate of the reliability 
of the composite score can be derived from the matrix in Table 12 by assum- 
ing that the lower bound of the reliability for each subscore is the highest 
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correlation that it has with any of the other subscores, then applying an 
extension of the Spearman-Brown formula. From this, we find that the reli- 
ability should be at least 0.71 which would make the standard error of 
measurement less than 5 PIF composite score points. 

Validity of the PIF c ompos i te score . A partial quick check, in the 
absence of time to conduct a large-scale validity study, was conducted by 
selecting 30 programs, 10 each with PIF composite scores of 30, 40, and 50. 
The documentations for these 30 were randomized, and each of four AIR eval- 
uators, independently and without knowledge of the PIF composite scores, 
rated each program on a three-point scale {3=high, 2=middle, l=low), solely 
on the basis of the documentation. Table 13 shows the interrelationships of 
the AIR ratings and the PIF composite score. 

TABLE 13 

Intercorrelations nf Fvalnators and PIF Composite Score (CS) 



Rater 


1 


2 


3 


4 


CS 


1 




.59 


.35 


.73 


.35 


2 


.59 




.63 


.68 


.55 


3 


.35 


.63 




.50 


.40 


4 


.73 


.68 


.50 




.59 


Composite Score 


.35 


.55 


.40 


.59 





If correction for restriction of range only is made, the lowest cor- 
relation of Rater and Composite Score (.35) becomes .48, and the highest 
(.59) becomes .67. The validity of the sums of all ratings with the PIF 
composite score was .58, depressed by course grouping (three-point scales), 
and also by speeded rating. Nevertheless, this correlation is significant 
at the 5% level. Thus, the agreement between the PIF composite score and 
rapid rater judgement is significant. 

Specific contributions to rejection . Low PIF composite scores resulted 
from unsatisfactory answers to important questions on the Program Information 
Form. For example, of 728 respondents: 

58% had unpublished reports or none 
43% had evaluation data for a year or less, or none 
62% did no tests of significance, and a further 14% found insufficient 
significance (Some respondents probably called simple comparisons 
"tests of significance.") 
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Of the 24% who apparently djd_ find significances at the 5% level or 
better, a surprisingly large proportion were to make only the 
completely trivial finding) that posttest results were better 
than pretest results, i.e., that some growth of unknown source 
had occurred 

69/0 did not calculate standard deviations or variance 

26% did not even calculate means or medians 

60% took no account of attrition (loss of sample) 

17% ignored "practice effect" and gave the identical test for both 

pretest and posttest—often 3 or 4 month*, apart 
41% achieved their results, in part at least, by spending 8 hours 

or more per week on the program 
14% achieved results by spending more than $300 extra per pupil per 

annum 

18% did not use nationally standardized tests 
61% had unsatisfactory information about the reliability of tests 
for their samples 

The mean PIF composite score was 36, with a standard deviation of 9, 
All available documentation was eventually examined for every program with 
a composite score of 30 or more (787, of the total). Many had no documen- 
tation at all; most of those that did confined it to descriptions of the 
program, and included local newspaper reviews. A large proportion of 
those that did try to present data used simple frequency counts only, often 
reporting the softest observations such as: "Before the start of the 
program X teachers considered progress of pupils satisfactory; afterwards 
Y teachers did so." 

Evan when real data was collected and treated statistically, a mere 
test of significance of gains between pretest and posttests was considered 
adtquate proof. There were obvious arithmetical errors. For example, in 
one case the sign was lost, and a loss over the treatment period was called 
a gain. 

Several programs sought help from outside consultants, not always with 
profit. In one such case, the consultant found that in 20 out of 21 classes, 
posttest means exceeded pretest. The actual gains were minute, the largest 
being less than a tenth of a standard deviation. However, the consultant 
obscured this by testing the frequency of 20 out of 21 on the chi -square 
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test, and showed that this table was highly significant. 

One consultant put wrong data through the computer, resulting in a 
demonstration of the grossest possible lack of matching between ostensibly 
matched experimental and control groups. The table reached us with incom- 
patible discursive treatment of the data, and the facts emerged as a result 
of an enquiry prompted by a puzzled phone call. 

Another consultant converted pretest scores to grade equivalents using 
the 1964 norms, but converted posttest scores on the 1973 norms. His erro- 
neous procedure led the consultant to claim that program participants had 
made subiitantial "gains." What the 1973 norms do show, however, is how far 
standards have dropped over the past 9 or 10 years. A performance that would 
have been equivalent to grade 1.0 in 1964 is still worth the same in 1973; 
children entering the system are as good as ever, and even a year later are 
still "par for the course." Thereafter, they would be credited with increas- 
ingly higher grade equivalents on this well-known test. A performance on 
Reading Comprehension at grade 3.0 in 1964 would get a grade equivalent of 
3.6 in 1973; 4.0 in 1964 would be a 4.6 in 1973; and 5.0 in 1964 would 
become 5.7 in 1973. By the tiniC grades 8 and 9 are reached, the difference 
is more than a full year. 

Samples of rejected prog rams with PIF com posite score s of 36. Brief 
descriptions of five programs with PIF composite scores of 36, the mean for 
the entire set of 728 programs, are presented below. The five programs 
were randomly drawn from the 26 with a composite score of 36. Only one of 
these five sent any additional evaluative documentation, and two sent very 
brief descriptions of the program. 

Case Com ments 

1 Had no standardized tests in 1972 because of lengthy teachers' 
strike and could show no evidence of gains on the criterion, 
and of course no statistical analysis of any sort. 

2 PIF responses indicated "Mean gain for less than one year 
is binger than expected," but "No significant differences 
found^'yet." Program takes about 10 to 12 hours per week. 
No allowance made for biases due to attrition. Descriptive 
material and two tables of data; no sample size. 

3 First grade gains were significant between 5% and 10"' one- 
tailed levels. No significance tests for other grades. 
Program takes 10 to 12 hours per week. No account taken of 
bias due to attrition. Comparison with norms. 
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kise. Comments (continue d) 

4 Not known whether program will continue for 2 more years. 
With regard to kind of improvement or gain, "Information 
not known." Significance tests were said to have been ap- 
plied, but no significant differences found yet, and no 
answers given about any other statistical summaries. Reli- 
abilities of tests "not yet determined." 

5 Evaluation reports "available but not published." Program 
in operation for less than 'c years. Significance between 
555 and 10% one- tailed levels. Reliability of tests not 

yet determined. No account t iken of biases due to attrition. 
Comparisons with norms. 

Samples of rejecte d programs with PIF composite scores of 50 or hig her . 
There were 64 programs with PIF composite scores of 50 and up. A random 
sample of 10 was drawn, of which one had been selected for packaging, and 
two for the catalog. One of those cataloged was rejected for packaging 
only for lack of detailed comparison and checking of its control group. There 
would have been difficulty finding a suitable control group because it was 
for vocational students. Apart from this lack, it was very well done, having 
one of the highest PIF composite scores (60) with correspondingly high values 
for each of the subscores. The other program that was cataloged had a PIF 
composite score of 51, Criterion 89%, Experimental Design 83%, but Statis- 
tical Adequacy of only 50%. A small amount of supporting evaluation was 
sent, though it was somewhat primitive (bar-graphs only) and inadequate. 
Presented below are brief descriptions of the other seven programs, with 
reasons for rejection. 

Case ^ Comments 

1 PIF composite score 50. Criterion 89%, Statistical Adequacy 
50%, Experimental Design 83%. Good materials, good reporting 
and evaluation, but data inconclusive due to usual problems 
of lack of satisfactory baseline. A promising program for 
re-examination in the future. 

2 PIF composite score 50. Criterion 78%, Statistical Adequacy 
75%, Experimental Design 72%. No tests of significance; gain 
of a quarter of a standard deviation. 

3 PIF composite score 54. Criterion 78%, Statistical Adequacy 
75%, Experimental Design 100%. Sent no supporting data or 
evaluation; included a paperback book instead with only 
tenuous relevance to the particular site. 

4 PIF composite score 51. Criterion 89%, Statistical Adequacy 
63%, Experimental Design 67%. Little data and no analysis; 
gains in standard deviations too smalt. 
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Case Comments (continued) 

5 PIF composite score 51. Criterion 100/', Statistical Adequacy 
63:.'., Experimental Design 94?'. No tests of significance ap- 
plied yet, as data not complete. Another program that should 
be re-examined when the opportunity presents itselr. 

6 PIF composite score 50. Criterion 78", Statistical Adequacy 
88';',, Experimental Design 83:".. No evaluation data or analysis 
sent. 

7 PIF composite score 52. Criterion 78. , Statistical Adequacy 
75", Experimental Design IT:.. No data, evaluation, or analy- 
sis sent. 

AJ R_ A'tj ff JLhj>d^ Sta^ S cre^^lT 3 

There were six evaluators, all with some background of statistical 
training and experience. At first, all programs with PIF composite scores 
of 40 or higher were examined by these evaluators, but later this was 
extended to include those with scores of 30 to 39. Before starting, the six 
evaluators met to discuss and establish guide rules. During evaluation 
there was also constant referral and consultation. For borderline cases, 
the telephone was used freely to obtain additional information or clarifi- 
cation. 

CI assif ic ation of reasons for rejection. Rejections were made because 
of serious flaws or lacks under one or more of the following headings: 

Su^ortiye do cumentat ion--A minimai requirement here was for some 
descriptive materials and evaluative data. Statistical analysis including 
significance tests was preferred, but failing this, some reference to testing 
and to accumulation of data was necessary. In fact, it was frequently neces- 
sary to compute elementary statistics such as the mean, standard deviation, 
and t^- tests from raw data. Computations reported were also checked for 
gross errors that were frequently associated with spuriously enhanced 
claims, but on occasion incorrect figures or formulae were applied to a 
program's disadvantage (e.g., using t-tests for uncorrected means, when 
the data were correlated). Gross arithmetical errors were not uncommon; 
one involved a reversal of sign converting a loss into an apparent gain. 

Programs were not followed up where the only material received, 
in spite of a request for supporting documents, was the completed PIF 
questionnai re. 
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Hypotheses tested — These could be explicit or implied, the only 
requirements being that they had to be relevant and focused appropriately; 
i.e., they had to concern cognitive achievement in reading, and should not 
be trivial. An example of a frequently found but trivial hypothesis tested 
by innovators was that pupils achieve higher scores on posttests than on 
pretests. 

Test used— These could be nationally standardized or locally 
developed, but they had to test some aspect of reading achievement directly, 
and include normative measurement; i.e., they had to allow meaningful com- 
parisons of group performances. So-called "criterion referenced tests" 
were of little value unless norms had also been established. Measurements 
of affect, motivation, or attitude, while important learning outcomes, were 
of no persuasive value as central themes. 

Sampling — While sampling is practical economy, it must not preclude 
useful generalization, and it must be large enough to imbue the generaliza- 
tion with sufficient precision. The method of sampling used had to allow 
generalization to a usefully defined population. For example, if volunteer- 
ing was used to recruit groups, one is unable to generalize legitimately 
the findings to "all grade 3 students" without adding the restrictive quali- 
fication "...and who would volunteer to attend on Saturdays." 

Samples of 30 or smaller are borderline at best, and had the 
further handicap that the results of activities of one or two teachers only 
could be represented. These were found acceptable only if there were strong 
additional arguments. 

The chart on the following page was used as a guide to acceptable 
sampling practice. 
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Shaded areas represent unacceptable sampling designs, and partly 
shaded areas, those that might be acceptable c^epending upon specific provi- 
sioHb. White areas are acceptable. For example, ure of the complete available 
sLiident population (e.g., all grade 4 students in the district) or a random 
sample of these students could have compared performances against either a 
blocked matched group, or their own previous performance, or a random sample 
(as a control group), or previous classes in that grade, or national norms. 
They could not be compared with performances of students above or below cut- 
off points on a test, or of volunteers. Volunteers could be compared only 
with blocked matched groups, or with previous performances. 

Compari so n group --There had always to be a comparison group, 
explici* or implicit, or the program was rajected. T' comparison '^roup 
had to be a reasonable "competitor." More or less -.rbitrary setting of 
achievement of criteria (as in "criterion referenced tests") is no substi- 
tute for a comparison group. This v/as a frequently used device, but in 
the absence of additional nor.native itieasuremerit, it was always rejected. 
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Equally unsatisfactory was the common procedure of setting up a 
straw man as a comparison group, as for example when the only comparison 
was posttest with pretest, or when the control group was unduly handicapped 
by a poor alternative program, or by conditions outside the educational 
environment. In some cases, sophisticated respondents used covariance analy- 
sis in an attempt to compensate for gross inequalities between control and 
experimental groups. Covariate adjustments are legitimate only where rela- 
tively minor discrepancies occurred after reasonable attempts had been made 
to match the contrast groups. One such rejection involved a case where a 
full standard deviation separated mean pretest performances of the two 
groups. 

Use of national norms was acceptable as long as ^ome attempt was 
made to demonstrate that the experimental group was reasonably normal. In 
fact, this is the only device open to a large number of district-wide pro- 
grams, and unfortunately several were unable to provide the additional 
demonstration needed, and through no fault of theirs. 

Data collection -- There are several ways in which faults in data 
collection can nullify a demonstration. For example, especially when the 
treatment period was short (often as little as 4 months), discrepancies 
between testing dates of experimental and control groups can account for 
differences in gains. Or when the two groups consisted of only two or 
three classes each, casual approaches to time limits by even a single 
teacher could generate apparent differences in performance. For this 
reason, small samples with this design were regarded with added suspicion. 

Units of measurement werj not considered as critical. Respondents 
were free to use raw scores, centiles, standard scores, stanines, or grade 
equivalents, as long as the usage was consistent and referred to a single 
source or table. 

Control of confoundi nq --This was a prolific source of rejection. 
The worst of it is that sources of confounding often tend to be correlated 
with other sources of bias, and therefore cumulative. For example, absen- 
teeism ha-, a low but significant negative correlation with performance, so 
that missing data tend to be that of poorer performers; and poorer performers 
can sometimes find anonymity more easily in control groups, than under the 
spotlight of special programs. Sample attrition is often partly the result 
of itinerant Tamil ies, and itinerancy can be both pn effect and a cause of 
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poor home backgrounds. Studies were reported with up to 60X loss of sample 
between pretests and positests. Again, with small samples, the confounding 
effects of individual teaching styles can influence results markedly. 

Descriptive stati stics--Mi nimal requirements demandea at least a 
measure of central tendency (e.g., mean or median^ and^ one of scatter (e.g., 
standard deviation or interquartile range). When significances of gains 
were at stake, correlations, covariances, or at least individual differences 
were necessary. This last involved recalculations of significances. Several 
projects submitted raw data only, and when possible, AIR staff programmed and 
produced the needed statistics. 

Statistical ana lyses--Sophisticated analyses were not demanded, 
although several provided analyses of variance and of covariance. In fact, 
some of those who did resort to complex analyses could lay claim only to 
rather minor real gains, and it was found necessary to examine the processing 
carefully. In doing so, inappropriate or incorrect applications of techniques 
•*(£ven to gross blunders in arithmetic or selection data) were found. In one 
Ca^e, the wrong data had been computed. 

Several programs, understandably, sought assistance from outside 
consultants, but not always to their advantage. 

A simple but effective technique, too seldom seen, involved just 
two-way or contingency tables, with chi-squares or critical ratios of some 
sort. 

Inferential statistics--A crucial requirement was the inclusion 
of one-tailed significances at, or better than, the 5% alpha level. Tests 
used could be critical ratios (for samples larger than 100), t^-tests, F- 
tests, chi-squares, or one of the many so-called nonparametric tests, like 
the Wilcoxon or Mann-Whitney tests. Multiple t-tests were suspect and in- 
volved extra work in sorting out claims of effectiveness. 

Once ir a while, cases were found whose claims were improved after 
correcting their wrong use of t-tests for uncorrected means with a single 
group, though this could occur only where even the wrong test showed signif- 
icant gains. It is theoretically possible for some programs with real gains 
to have been rejected because of a failure to use the correct formula. 

Ed ucatio nal significance— Thi s was a wis3 requirement of both 
Right to Read Program staff and the ORP, and a principle with which 
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AIR evaluators have always concurred. Mere statistically significant gains 
are not enough; the gains must also be large enough to justify the effort 
and cost involved in the program. It is difficult to set a firm criterion 
here, but the somewhat arbitrary threshold of a gain of one-third of a stan- 
dard deviation appears meaningful. In practice, over a wide variety of sit- 
uations involving ratio measurement such as height, or weight, or lengths, 
one-third of a standard deviation seems to be, if anything, smaller than a 
just noticeable difference. In the behavioral fields, a shift in the per- 
formance of a group of one-third of a standard deviation may seem small, so 
this can be regarded as a lenient measure, and appears to be a standard 
adopted by the DRP as well. As further support for this limit, several of 
the best designed and controlled programs with high PIF composite scores 
produced gains of this order. 

The Dissemination Review Panel (DRP): Fourth Stage Screening 

The final approval for the dissemination of the 27 programs recommended 
for packaging by AIR rested with the Dissemination Review Panel (DRP) of the 
Office of Education. The DRP sets reasonable standards of sophistication in 
evaluation for programs approved by it for dissemination. In the case of 
this project, criteria established related to the same areas of achievement 
o>jtcomes as those applied throughout the successive screening stages. These 
were focused upon the contrast of gains for program participants with a non- 
program comparison group, statistical and educational signficance, general i- 
zability of the findings, and adequate size. In short, the main objective 
of the DRP is to evaluate all evidence regarding each program and to decide 
whether the totality of data indicates that the effect of the program is 
related to the program processes and cannot be equally justified by rival 
hypotheses. The basic question is quite simply for each review: Is this 
a program producing improved cognitive gains, that is worthy of endorsement 
by the federal government for replication by others? 

Statistics in relation to the DRP decisions . Table 14 identifies the 
27 programs recommended for packaging by AIR, and the results of their 
review by DRP. Of the 27 programs recommended by AIR, DRP approved 14 and 
action is pending on 2. Two of these were later rejected by Right to Read 
because they were not quite suitable for packaging, as they haa already 
received considerable and wide publicity. 
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The mean PIF composite score of those accepted by the DRP was 47.85; 
for those rejected, it was 42.50. The difference (5.35) was significant 
at between the ]7c and 2% levels (two-tailed), and yielded a biserial corre- 
lation coefficient of .28. The standard deviation of the scores for these 
was 6.1; for the whole group of 728, the figure was 9.0. Thus, the corre- 
lation obtained was for a selected sample; after correction for homogeneity 
it is .67 for the total group. It is pleasing to find this association 
between PIF composite score and DRP decisions, and it is an indicator of the 
validity of the PIF composite score. 

The most frequent objection raised by the DRP pertained to the nature 
of the baseline used, or to inadequacies about it. In one case, a school 
district had had no way of showing that gains were not the result of an 
influx of pupils of higher ability. In another, the comparison was only 
with results of an alternative publisher's program. Smallness of samples, 
or of numbers of independent sources of samples (e.g., classrooms), and 
smallness of gains with big samples were also naturally suspect. 

Conclusions and Recommendations 

Right to Read staff, the OE Dissemination Review Panel, and staff of 
the American Institutes for Research were substantially at one in the per- 
ceptions of the aims of the study. These were objectivity and rigor, and 
defensible standards for admissible evidence. Over the past decade, the ideals 
of our society, backed by record-breaking economic commitment, produced a fer- 
ment of innovation, in which practically any practitioner with a modicum of 
thrust could put long-held ideals into practice. Too often claims for success 
were later found to have been backed by inadequate evidence, by faulty logic, 
and by special pleading. The evidence presented in this report is to a large 
extent the outcome of a limitation in the expertise of educators. 

Most educators are not researchers, and there is no reason why, in 
general, we should expect them to be. Teaching is a full-time and demanding 
profession in its own right, with aims and ethics which are often incompatible 
with the aims and requirements of research. However, from the research view- 
point, the needs for strong inference demand that the alternative to an inno- 
vative program must be given a fair opportunity to demonstrate a superiority 
to the innovation. But for the teacher, no group of pupils can deliberately 
be put to possible disadvantage, so appearance of control groups is often 
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fortuitous. Or again, research design must avoid introducing more changes 
or variables than it can control for. But a good superintendent will allow 
only lack of funds, staff, or time to limit the number of "changes for the 
better" he can make. If he knows of a better testing program, he will 
introduce it at the same time as innovation in teaching. Many more examples 
could be given, but it all boils down to this: 

• Good teaching practice can mean poor research, and vice versa. 

• Quite normal educational circumstances can rob a sound innovation 
of the opportunity to prove itself. 

• Even with the best intentions, innovators will more often than not 
lack the research r.ophistication to collect vital data, or to plan 

a conclusive demonstration until it is too late 'to do so adequately. 

Educators must be given research assistance at the time they need it 
and only in the quantities that they need. Two important changes are called 
for: 

1 . Contact between innovator and researcher must be made in time 
for the latter to influence researc h design a nd collection of data before 
insurmountable obstacles intervene . This would mean that initial contact 
between innovator and researcher should occur at least 12 to 18 months before 
the crucial comparisons are to be made. It would involve no sacrifice of au- 
thority on the part of the innovator; he would be free, both to make his own 
suggestions for compromise, or to reject the whole investigation if he 
feels that it is becoming a handicap to his primary function. 

2. Evaluators need to take exigencies into better account . This 
does not mean lowering the standards of research rigor. If anything, experi- 
ence has confirmed the wisdom of the criteria set by all three investigating 
parties in this study. But special provision needs to be made for recogniz- 
ing a class of innovation which, while setting high standards of control, 
finds loopholes which it cannot close and therefore ends with an inconclu- 
sive demonstration. Perhaps this is what was intended by references to 
"promising programs" in the Right to Read request for proposal (RFP) for 
this project. If so, the issue needs clarification and ground rules formu- 
lated. Guidelines in this area will be difficult to develop, but the 
effort has to be made. Without some circumscriptions, and given only an 
overriding concern that deserving innovations may be ignored, future inves- 
tigators will be swamped with unprofitable demands on their time. With this 
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in mind, we make the following more specific recommendations. 
Specific recommendations . 

Contact between innovator and researcher should occur well before 
outcomes are to be assessed. This contact should start with 
advertising intentions and inviting applications. 

Applications meeting certain minimum requirements would be fol- 
lowed by correspondence and telephone conferences. Respondents 
would commit themselves only to discussions and planning. They 
should be free to withdraw at any point, and otherwise to over- 
ride advice— at the risk, of course, of having their program 
rejected from consideration in the end. 

A series of modules of instruction should be prepared on selected 
evaluation topics such as: 

• Benchmarks by which to assess educational progress, e.g., 
comparison methods 

• Practical guides to methods of sampling, with their uses 
and limitations 

• Choosing appropriate tests and measurement units 

• Sources of bias in results, and ways of reducing or 
compensating for them, e.g., missing data 

• Simple but effective statistics to use, with common sense 
interpretation 

• Common and uncommon pitfalls, with examples and explana- 
tions, e.g., regression to the mean 

• Profitable use of consultants, and when to use in-house 
expertise 

• Modern mechanical and electronic aids to educational 
demonstration 

• Pleading special cases profitably, e.g., small samples, 
large expenses, affective criteria, lack of controls 

• Adequate reporting 

Larger organizations and institutions would be free to use their 
own in-house experts o r to employ consultants. But funds should 
be available, particularly for small institutions, to have two or 
three visits by those who would be presenting the evidence eventu- 
ally. These visits would serve both to deal with special local 
problems and to keep the study as a whole in competitive form. 



- 52 . 62 



•• The use of local consultants should be detennined. There should 
be continuous communication between the Institution and the re- 
viewers. 

•• Lastly, there should be a special class of Innovation where normal 
demonstrations of success are precluded. But special cases should 
be kept to a minimum, and the reasons for the disability should 
have been clearly demarcated for 1 year before the final scrutiny. 
Moreover, there should be strong reasons other than statistical 
for a presumption of success. These should Include arguments under 
specified headings such as: 

• Clear formulations of objectives and rationales for 
procedures, together with a unifying philosophy 

• Clear demonstrations of substantial and sustained 
effort, such as preparation and use of materials, 
on-the-job training, community support, and so on 

• Clear economies in specified areas, e.g., of time, 
or of finances, or of personnel, without sacrifice 
of achievement 

• Systematic effort to collect hard data for future 
comparisons, and to avoid problems in this area in 
the future 

• Satisfactory replicability and exportabi 1 1 ty 

• Early identification of possible sources of con- 

• founding, with real provisions for their assess- 
ment (For example, it must be possible to elimi- 
nate, or at least to assess, the contribution to 
change of such possible explanations as change in 
the population, or contribution by individual 
teachers, or differences in tests used.) 

• Prime focus on cognitive, normative change 

In summary, if in the end claims are to lack the support of hard 
data, it should be unavoidable, and recognized at least a year 
before the final scrutiny. None of the other criteria laid down 
by Right to Read should be sacrificed. This perhaps will give 
effect to the spirit behind the Right to Read concerns about "prom- 
ising programs," without Injustice to those programs who had to 
pass the more stringent controls. It would make it possible to 
apply more objective guidelines, with better control of the stan- 
dards of those so selected. 
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Chapter V 
DISSEMINATION PRODUCTS 



Two major dissemination products were developed as a result of the 
search and screening operations. These were (1) a set of packages present- 
ing multi-media descriptions of exemplary reading programs, and (2) a cata- 
log of reading program descriptions. Also, a guiderule that conveniently 
summarizes 17 items of information for each of the packaged programs was 
developed. 

Packages 

The primary criterion for package development was that the only pro- 
grams packaged be those approved by the Office of Education Dissemination 
Review Panel (DRP) as demonstrating exemplary status. A program package 
was developed for each of 12 of the 14 reading programs approved by DRP. 
Development of each program package required the assembly of information 
into a framework that described major program elements or components. This 
information was partly obtained from the PIF, from available program descrip- 
tion and evaluation documents, from telephone consultation with program 
staff, but primarily through a 5-day visit at each reading program site. 

Specifications for basic program information . The basic program 
information specifications for each package are contained in the Inventory 
of Program Data (IPD) shown in Appendix E. Each item on the IPD focuses 
on processes and aspects of a program that might be necessary to repeat if 
it were to be implemented at another site. The format of the IPD provides 
for indicating, at any point in the data collection process, information 
that is still needed about the program. 

From items on the IPD, outlines were prepared to guide authors of the 
program handbooks for the packages. The IPD reference column in the 
right margin of these writing guides indicates how each section is keyed 
back to the IPD. The purpose of developing outlines and instructions keyed 
to the IPD was to ensure that authors would prepare program handbooks that 
were adequately comprehensive for use by potential replicators. The Outline 
for the Instructional Handbook and the Outline for the Program Management 
Handbook are also included in Appendix E, following the sample IPD. 
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DescrUtion of package components . Each program package was planned 
to consist of a filmstrip with audio-cassette script, a handbook describing 
the program management components, a handbook describing instructional com- 
ponents, and typically, two charts--ont. a flow chart depicting program pro- 
cess and the other, an objectives-activities-assessment chart. Thus, there 
were two central target groups for the packaged information: first, educa- 
tional managers such as superintendents, principals, and reading program 
directors; second, instructional staff. However, the packaged materials 
are also adaptable for a wider audience (school boards, parent-community 
groups, students). Flexibility of use is built into each package component, 
as summarized below. 

• Filmstrip— The filmstrip for each package serves multiple purposes. 
It is not strictly a simulated site visit, but must, nevertheless, depict 
major program highlights and essential features. It certainly serves as a 
quick presentation of the total program, enabling a viewer to focus on the 
program's central thematic elements and to view classroom activities. 
Although a filmstrip with audio commentary cannot detail the many specifics 
and the complexity of any reading program, it can motivate viewers to examine 
the detailed information summarized in the two handbooks. Furthermore, the 
filmstrip itself can also succinctly summarize program features for presenta- 
tion to school and community groups whose endorsement and support is necessary 
when planning potential implementation. 

• Ma nagement Handbook --This handbook contains information about how the 
program was planned, implemented, managed, budgeted, and evaluated. Essential 
management and staff training techniques are included. Cost dat5 provided 

by programs are reported, but recognizing that new sites will have different 
resources available, different salary scales, different administrative 
structures, different local accounting practices, and different numbers of 
students to serve, the emphasis is on budget options that may suggest other 
ways of providing the required resources. 

• Instructional Handbook— In the Jns trwcti ona? Handbook, information 
about program activities is first summarized quite briefly (Program Overview), 
then expanded upon (Program Activities), and then selectively detailed 
(Specific Examples of Instructional Procedures). In this way, teachers have 
options for selecting the level of detail that serves their various purposes 
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In reading this handbook. If they are scanning several programs or 
approaches very superficially, they need read only Program Overviews, 
If they are car :fully comparing options, the Program Activities section 
will thoroughly acquaint them with classroom activities. If they are using 
the program or some of Its components, the chapter on Specific Examples of 
Instructional Procedures will be a valuable, practical source of selected 
details of program operation. 

• Charts— Most packages contain two charts. A flow chart depicts the 
program process. The specific nature of the Illustrations varies according 
to program characteristics. For example, a simulated classroom layout 
showing a sequence of activities for a single child may be appropriate for 
depicting one program, but a graphical representation of Instructional 
procedures or program structure may be more appropriate or effective 
for another. The second chart has three columns In which are summarized 
the program's objectives and the activities and assessment techniques 
related to each. Together, the two charts present the program In capsule, 
form, and may be useful devices In program orientation workshops or for 
quick reference. 

The Catalog of Reading Programs 

A catalog was prepared consisting of 222 one-page descriptions of 
reading programs. The purpose of the catalog Is to present an overview of 
promising practices throughout the nation In the field of reading Instruction. 
Programs in the catalog are grouped into four general areas: Elementary 
Programs, Secondary Programs, Adult Programs, and Special Programs. Taken 
together, the 222 program summaries exhibit an Impressive array of approaches 
for many target groups. 

It must be emphasized that the descriptions presented In the catalog do 
not Include programs endorsed as exemplary by the Office of Education. How- 
ever, In selecting programs, and consistent with practical time limits for 
the toelopment of the catalog, a single scale was established based upon 
PIF Hems to determine candidates for catalog Inclusion. Thus, programs whose 
descriptions appear In the catalog do represent stronger evaluation components. 
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The selection scale was based upon unit weighting of each of 10 PIF items 
as follows: 



Item 6 



Instruction focused upon cognitive improvement 



Item 7 



Program in existence for at least 1 year 



Item 8 



Expectancy of at least 2 years' continuation 



Item 9 



Evaluation reports available since 1968 



Item 12 



Participant size of 30 or higher 



Item 13 or 14 Evidence of success based upon reading tests 



Item 20 



Up-to-date program description available 
Per-pupil costs below $500 



Item 28 



Item 44 



Baseline reading skills measured routinely 



More than 90% of the programs in the catalog had PIF composite scores above 
the mean of 36 points. None of the remainder had scores of less than 33. 

Catalog descriptions were confined to a single page and organized 
according to common headings: Program Size and Target Population; 
Year Started; Staff; Major Features; Facilities, Materials, Equipment; 
Cost; and For Further Information. Instructions for preparing catalog 
descriptions were developed for authors based on the unshaded IPD items 
(see IPD in Appendix E). All writers were required to adhere to the one- 
page limit and standard section headings. However, they were allowed to 
adapt the IPD-based instructions for preparing these summaries to suit 
program emphasis. This flexibility allowed them to convey to readers 
each program's unique character. A sa* pie program description from the 
Catalog of Reading Programs is shown in Appendix F. 

Product Review Procedures 

All products, when prepared, were reviewed by relevant parties and 
when necessary, by required agencies. 

Filmstrips and scripts for each program were prepared by AIR staff 
on the basis of site visits and consultations with reading program staff. 
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Filmstrips were reviewed by a task force composed of National Right to 
Read Program staff and revised if necessary. Finally, approvals for 
the filmstrips and audio commentaries were obtained from the Office of 
Public Affairs of the Office of Education whose representatives viewed 
all filmstrips prior to final production. Filmstrips were also reviewed 
by two members of the AIR Advisory Panel, Dorothy Gaither and Ruth 
Hessenflow. 

All printed materials (handbooks and charts) in each program 
package were forwarded in draft form to reading program staff. Based 
on feedback from their review, revisions were made. Revised handbooks and 
charts for several programs were also reviewed by Dorothy Gaither and Ruth 
Hessenflow. The National Right to Read Program staff was sent copies of 
the final versions of handbooks prior to production. 

Packa ges Proail^d 

Packages were produced for the 12 programs listed and summarized in 
Table 15. As a group, the 12 programs exhibit a range of approaches, yet 
all are directed toward improving reading skills reaardless of the age or 
circumstances of their participants. 
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Guiderule 

One additional dissemination product was produced which summarizes in 
a single circular rule 17 items of information for the 12 packaged programs. 
This guiderule permits one to quickly and conveniently obtain information 
relating to the following items for each program: 



(1) # Students 

(2) Program Draws Students From 

Inner City 
Urban Residential 
Suburban 

Small Town or Rural 

(3) Program Setting 

Publ ic School 
Private School 
Community 

Correctional Institution 
Other Agency 

(4) Age/Grade Level of Students 

(5) Materials Cost for Group of 30 

(6) Program Duration 

Periods (hours) 
Days/Week 
Semesters 
Years 

(7) Time Spent in Grouping Patterns 

Individual 
Small Group 
Large Group 

(8) Program Emphasis 

Readiness 
Beginning Reading 
Remedial 
Accelerated 
Teacher Training 
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(9) Special Resources Required 
Visual Equipment 
Audio Equipment 
Library or Resource Center 
Museum 



(10) Major Groups Included 

Black 

Indian 

Oriental 

Spanish-Speaking 
White 

(11) Special Parent Roles 

Aides 

Home Teachers 
Advisory Committee 

(12) Source of Materials 

Commerical 
District-Developed 
Teacher-Made 
Student-Originated 

(13) Hours of Inservice Required For 

Teachers 

Paraprofessionals 

(14) Staff Requirements 

Teachers 
Teaching Aides 
Clerical Aides 
Specialists/Consultants 
Evaluators 

(15) Tests Used 

Achievement Pre - 
Achievement Post - 
Ability 
Diagnostic 

Criterion-Referenced 



(16) Instructional Strategy 

Decoding 
Basal Reader 
Language Experience 
Linguistic/Oral 
Programmed Learning 

(17) Number of Sites Where Program is Operating 
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Diffusion Planning: A Look to tiie Future 



The efforts of this study have produced 12 information packages for 
validated, exemplary reading programs, and a catalog of over 200 additional 
program summaries. The information provided by the packages is for use by 
staff at potential sites in deciding whether or not to copy all or part of 
a program and in planning and implementing changes that are decided upon. 
The catalog contains one-page summaries of reading programs that, taken 
together, provide potential users with many ideas for upgrading their local 
reading practices and approaches. 

Merely producing these materials will not reduce America's reading 
problem. Nor will the packages or catalog alone cause anything new to 
happen. These products must be used in conjunction with other elements in 
a sound field test plan in order to bring about change. 

Once the packages ara available, the problem is two-fold: 

• Disseminating packages to sites that need and want to improve 
their present practices in reading instruction 

• Determining the kind and amount of assistance a site needs in 
order to use the package to select, adapt, implement* and 
evaluate the exemplary program or parts of it 

An adequate field test plan must provide solutions to both aspects of the 
problem--dissemination and implementation--if the packages are to have the 
desired impact on local reading curricula. 

Therefore, systematic pilot study planning would need to: 

• Identify specifications for sound diffusion 

• Design an adequate field test plan according to local site 
specifications 

• Provide needed assistance to potential users 

Assumptions underlying diffusion planning . Identifying the specifi- 
cations for diffusion planning is based on important assumptions. One 
assumption is that the packages are more likely to promote improved reading 
instruction and reading achievement if they are part of a delivery system. 
The delivery system is made up of several components, only one of which is 
the pacKage material. Other components of the delivery system must provide 




for identifying potential users and supplying them with the packages, and 
bring to these potential ui.ers the support services they need to success- 
fully select, adapt, implement, and evaluate the exemplary reading practices 
or approaches detailed in the packages. Similarly, the catalog is most 
likely to lead local staff to implement suitable modifications in their 
reading programs if it is part of a delivery system that prompts potential 
users to do this. 

A second assumption underlying diffusion planning is the package con- 
cept. This concept grew out of the criteria established cooperatively by 
Right to Read and AIR for identifying exemplary reading programs. The 
package concept is important because it also reflects Right to Read's broad 
goals for the study. These goals were to prepare detailed descriptive 
materials that would faithfully document important features of the exemplary 
programs--features that could conceivably be crucial for successful imple- 
mentations at other sites. Consistent with this emphasis, the package 
concept has these major features. 

• Only proven successful programs have been packaged. 

• Package materials are faithful to the program as implemented 
at the successful site. 

• Package materials describe processes used at the site objectively 
and literally so that new sites can copy these successful models. 

• Package materials descr^bu essential elements of program opera- 
tion in different degrees of detail. A filmstrip and wall charts 
communicate program highlights. Handbooks for teachers and 
managers provide program overviews and detailed procedures. 

• By supplying these levels of detail, potential users have 
options for selecting the amount of information that serves 
thair various iburposes in reading or viewing these materials-- 
e.g., scanning, comparing, or using the program or some of its 
components. 

•Users are viewed as active change agents, not as passive recip- 
ients who plug in the package and wait for results. Thus, 
the package materials do not present descriptions in a tightly 
prescribed, step-by-step format that might alienate or be ignored 
by the user. 
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• To further avoid giving the impression that exact-copy replica- 
tions are expected at new sites, the handbooks empha^'ze pro- 
visions the program makes for ongoing modifications by teachers 
and managers. Examples of changes made and their rationale or 
justification are given to emphasize systematic approaches to 
adapting the program to changing needs. 

Elements of diffusion planning . The assumption underlying product 
development in this study is that no dissemination materials are teacher- 
or people-proof. The packages and the detailed information they provide 
for teachers and managers should be regarded as one component of an adoption 
plan. To increase the likelihood of successful implementation, they should 
be used in conjunction with other components of the plan. Diffusion 
planning consists of several elements: 

• Objecti ves- -The broad objective of dirfusing information about the 
exemplary reading programs is to prompt changes in local programs 
that will result iti improvements in reading and reading instruction. 
Diffusion planning therefore focuses on where this change is to 
occur, in this case local sites. The focus on local sites means 
that diffusion planning will have to identify the kinds of local 
needs, and the means for check-listing them, that potential users 
must examine while they review the package materials. One element 
of diffusion planning, then, is to consider the means for encourag- 
ing potential users to match options (exemplary programs) to local 
needs, preferably based on structured self -assessments of the 
strengths and weaknesses of their existing programs. 

• Change agents- -Change agents and change plans need to be defined. 
This includes identifying a delivery system i^s part of the change- 
ugent network. The main change agents will be those dealing with 
potential adopters at local sites. Questions to be examined are 
the one-to-one matching of roles of the change agents with the 
adopters (managers, teachers), and the role of national, state, and 
regional agencies in supporting diffusion and implementation. The 
purpose of this activity will be to define specifications for 
designing an adequate implementation system. 
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• Materials --Still another element of diffusion planning is speci- 
fying those materials that will be needed to help users select, 
adapt, implement, and evaluate the exemplary programs, practices, 
or approaches. The packages are designed to be a critical part of 
the diffusion system, but they are not spontaneously replicable or 
adaptable. Other materials to be considered would include news- 
letters to publicize the packages, convention or conference sym- 
posia to exDlain the search, selection, and packaging rationale, 
preparation of workshop materials to diffuse change information and 
strategies through the Right to Read state networks, and needs 
checklists for use by adopters to identify technical assistance and 
other needs. 

• Context of adoption --Diffusion planning will also consider the con- 
text for bringing together the potential users, the delivery system, 
and the materials discussed above. The definition of context must 
account for local needs and information about unique student, in- 
structor, and local variables that could obstruct or facilitate 
changes in existing reading programs. The purpose of this activity 
will be to set the stage for generating a realistic field test plan-- 
one that can accommodate the environment or constraints in which 
change must occur. 

• Evaluation- -F1nal ly , diffusion planning will involve some pre- 
planning for evaluating the effectiveness of the dissemination and 
implementation strategies and results. It would be exoected that 
these preliminary plans will be modified and exoanded when the field 
test is implemented. 



A Field Test Plan 

Assumptions underlying a local field test plan . A local field test 
plan will reflect the following assumptions: 

• The packages and the catalog are more likely to promote improved 
reading instruction and reading achievement if they are part of 
a delivery system. 

• The packages and the catalog are major components of their 
delivery systems, but other components must bring needed support 
services to users and prompt them to change local educational 
policy and practice. 
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• The delivery system should not compromise the original package 
concept; the packaged programs are by definition proven successes 
and should be presented faithfully. However, the field test plan 
should encourage flexibility on the part of potential users in 
adapting these programs to local needs. 

• The field test plan should recognize that nearly all of the 
changes made as a result of diffusion will be adaptations. 

The field test plan should recognize that potential users are 
active change agents and should provide them with a framework 
they are free to use in order to systematically adapt the exem- 
plary program or portions of it to local needs. 

Components of a local field test plan . The components of a local field 
test plan should include, but not be limited to, the following: 

• Change agents . Who are they? At what level do they operate? 
What are their roles? 

• Context of adoption . How will sites be selected? How will key 
site personnel be identified? What are local variables that may 
facilitate or block diffusion and/or change? 

• Change strategies . How will packages be brought to potential 
use^s? What services must be supplied to facilitate selection, 
adaptation, implementation, and evaluation of the new program or 
approach? Will there be a need for workshops or other hands-on 
experiences in which the diffusion team and the user are brought 
together? 

• Packages plus supporting materials . The packages are viewed as 
a major component in the delivery system. However, other mater- 
ials will be developed to support the services required by the 
diffusion implementation system. Materials could conceivably 
include needs-assessment checklists that will help potential 
users specify the additional information and assistance they 
will require to proceed with imp''ementation and evaluation. 
Training materials for workshops on planning the program imple- 
mentation and evaluation will also be developed. (See Evaluation 
below.) Another category of materials will be tho^e designed 
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to publicize the packages in order to attract potential users. 

* Evaluation . This component of the delivery system provides 
technical assistance new sites need to develop sound evaluation 
designs. This may require that a special training package on 
evaluation be developed. The package would be aimed at personnel 
in typical school districts or other agencies, people who impla- 
ment programs and must evaluate, but do not have a research ex- 
pert's expertise in evaluation. The purpose of providing users 
with help in designing evaluations before they implement programs 
is to lay the foundation for eventually assessing the success of 
the program. It is recognized that evaluation of student achieve- 
ment outcomes will not be appropriate or valid until the shake- 
down period experienced by newly installed programs has oassed. 

• Costs . A field test budget covering costs for activating the 
field test plan will need t^. be developed. Included will be 
costs of technical assistance in implementing and monitoring the 
field test and in evaluatir its outcomes. Also included are 
associated costs for staff and materials. 

Early, close coordination with the reading program staff at each of 
the 12 sites will be necessary in order to develop a sound field test plan. 
These sites are not necessarily equipped to offer exactly the same imple- 
mentation support to new sites nor will the same diffusion format always 
be appropriate in each case. A separate technical report has been prepared 
which discusses these components in greater detail, based on the practical 
experience of implementers who have sought to foster successful replications 
or adaptations of innovative educational programs. 
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Criteria Sheet 

Cover Letters: 
Ven>ion 1. Program Nomination Panel 
Version 2. S^ate Level Nomination Sources 
Version 3. Large City Superintendents 
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Other Groups 
Version 4. Adult/Special Education Contacts 
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Offlci of Managimint and Budgrt 

No. 51 - S7303: 
Approval Expires 31 Doctmbar 1973 



PROGRAM NOMINATION FORM FOk 
OUTSTANDING READING PROGRAMS 

******** PLEASE USE A SEPARATE FORM FOR EACH! PROGRAM NOMINATED 
i. Name of Program 



********** 



2. Program Director Title 



Address Phone 



3. If Program Director is unknown to you, please give the name of contact 

Name. 

Address Phone 



4. Type of school or organization in which th3 program operates 

Pre-School EMH 

Kindergarten Bilingual 



Klementary School Adult Program 



High School Other (Describe) 

College 

5. If the program operates in a school, please give the name of the District 
Superintendent. 



Name 



Address 



6. What evidence, sign, or indication qualified this oronram as a success in 
your opinion? 



7. What are the majjr features of this program that contribute to its success 
in your opinion? 
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(over) 



8. Are you aware of any documented, published, or unpublished evidence of 

effectiveness describing the program or its effectiveness? (Please provide 
complete references if known.) 



Nominators 's Name 

Address 

Phone 

Date 



RETURN TO: 

Right to Read Project 

American Institutes r Research 

P. 0. Bnx 1113 

Palo Alto, Calif. 94302 
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American Institutes for Research 
Palo AUo, California 
Right to Read Project 

CRITERIA FOR SELECTION OF OUTSTANDING READING PROGRAMS 

Populati on 

Participants may represent any grade level from preschool through senior 
high school, adult and community programs, and participants from programs 
for special target populations, e.g., prison inmates, Head Start children, 
handicapped students, adult illiterates, etc. 

Size 

The program, in the case of a school district not concerned with special 
tarqet populations, wmt be used throughout the school building with all 
children at a given grade level or cluster of grades. 

Innovativeness 

The program may include practices recognized as new, creative, or unusual 
as well as traditional practices which are producing exceptional results. 

Exportabi 1 ity 

Tho various instructional materials, activities, services, organizational de- 
tails, and key procedural information are amenable to wide-scale dissemination. 

Location 

The program operates within the United States, its possessions or territories, 
or dependent schools. 

Long evi ty 

The exemplary practice or program has been in operation for at least one year, 
and there is no reason to expect that it will not continue for another two 
years (through 1974-75). 

Evidence 

Achievement in reading-related skills has been substantially improved as a 
result of the program. 

Recency - Recent evaluation data are available (since 1 January 1968). 

Avai labi lity - Descriptive and evaluative reports are ava1lat)lp. 

roinplete ness - SufflrUMit Intormation is availabU^ to evaltliitu thu |'ronr^li\» 
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VERSION 1. Program Nomination Panel 



AIR 



AMERICAN 
INSTITUTiS 
FOR REIIARCH 



P.O. Box 1113 t Palo Alto. California 94302 • (418)493-3550 • Cable; AIRESEARCH 



The National Right to Read Program of the S. Office of Education 
has selected the American Institut^B for Research to Identify up to 25 
outstandingly effective reading programs that are nov In use In Amerlcaii 
schools and other Institutions and to publicize these programs to other 
school systems and institutions th^t may wish to adopt them. 

You have been selected as a person knowledgeable about Innovative and 
exceptionally successful reading programs. We hope that you will be 
vllllng to participate in this study by nominating one or more reading 
programs that you know to be successful as candidates for inclusion in 
the group to be publicized and ''packaged** for distribution. We are 
specially concerned about the process which will create a setting for 
successful output from this project* **Packaging** in the sense we are 
using it is more broadly conceived than is generally interpreted* The 
**packages** from this study will stress the total comprehensive planning 
activity necessary to successiully duplicate the program elsewhere* but 
will not Include the specific instructional materials used by students 
and teachers in the classroom* Accordingly, I am enclosing several 
Program Nomination Forms* If you should need additional copies of the 
form, I shall be glad to send them* 

The program or programs that you nominate may operate at any level from 
preschool through adulthood* A program^ for our purposes, is defined, 
in the case of a ochool district, as one which is used throughout the 
school building at a given grade level, in a cluster of grades, or for 
a special population (such as a program that is being implemented through- 
out the primary grades or in all eighth grades or for all deaf children) « 
An effort leveled at only one fifth-grade class where there are other 
fifth-grade classes would not qualify as a program* Neither would the 
efforts of a single teacher who is **very successful in teaching reading** 
qualify as a program. Very large units, such as programt being imple- 
mented statewide, are of interest to us and may be nominated for con- 
sideration. A program, like a package, is conceived of in a global 
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An Equal Opportunity Employer 
A-4 



2 



sense and includes all those things necessary to Implement it success- 
fully • They may be aimed at special subgroups at any of these age levels. 
We are interested in programs funded by local tax sources and private 
sources, e«g., business, industry, foundations, etc., as well as those 
specially funded by states and the federal government, 

A list of criteria for selection of outstanding reading programs is 
enclosed. If you think a program might meet these criteria but are 
not sure, please nominate it anyway. If yoxi would like to nominate a 
program which does not yet have statistical evidence of success, do so. 
In either case, please give particularly careful thought to your answer 
to Items 6 and 7 on the Program Nomination Form. Both items are designed 
to identify reasonable, verifiable proofs of success for nominated pro- 
grams. Evidence may include — but need not be limited to — statistical, 
''hard" evaluation data; however, such evidence must be clearly defined. 

Since we are working on a very tight time schedule, we voMld appreciate 
a response from you at your earliest convenience. In order for us to 
make the appropriate contacts with the schools you nominate, we must 
receive word prior to November , sooner if possible. 



Cordially yours. 




Project Director 
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VERSION 2. State Level Nomination Sources 




AMERICAN 
INSTITUTES 
FOR RESEAf CH 



P.O. Box 1113 • Palo Alto, California 94302 • (415) 493-3560 • Cable; AIRESEARCH 



The National Right to Read Program of the IJ. S. Office of Education 
has selected the American Institutes for Research to identify up to 25 
outstandingly effective reading programs that are now in use lii American 
schools and other Institutions and to publicize these programs to other 
school systems and institutions that may wish to adopt them. 

You have been selected as a person knowledp;eable about innovative and 
exceptionally successful reading progrwis. We hope that you and your 
staff will be willing to participate in this study by nominating one 
or more reading programs that ycu know to be successful as candidates 
for inclusion in the group to be publicized and "packaged** for distribu- 
tion. We are specially concerned about the process which will create a 
setting for successful output from this project. ''Packaging** in the 
sense we are tio ing it is more broadly conceived than is generally 
interpreted. The "packages" from this study will sttess the total com- 
prehensive planning activity necessary to successfully duplicate the 
program elsewhere, but will not include the specific instructional 
materials used by students and teachers In the classroom. Accordingly, 
I am enclosing several Program Nomination Forms. If you should need 
^daitional copies of the form, I shall be glad to send them. 

The program or programs that you nominate may operate at any level from 
preschool through adulthood. A program, for our purposes, is defined, 
in the case of a school district, as one which is used throughout the 
school building at a given grade level, in a cluster of grades, or for 
a special population (such as a program that is being implemented through 
out the primary grades or in all eighth grades or for all deaf children). 
An effort leveled at only one fifth-grade class where there are o^her 
fifth-grade classes would not qualify as a program. Neither wotild ' 2 
efforts of a single teacher who is "very successful in teaching rea ng** 
qualify as a program. Very large units, such as programs being imy 
mented statewide, are of interest to us and may be nominated fc ct 
sideratlon. A program, like a package, is conceived of In a globax 
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sense and includes all those things necessary to implement it success- 
fully. They may be aimed at special subgroups at any of these age levels. 
We are interested in programs funded by local tax sources and private 
sources, e.g., business, industry, foundations, etc., as well as those 
funded by states and the federal government. 

A list of criteria for selection of outstanding reading programs is 
enclosed. If you think a program might meet these criteria but are 
not sure, please nominate it anyway. If you would like to nominate a 
program which does not yet have statistical evidence of success, do so. 
In either case, please give particularly careful thought to your answer 
to Items 6 and 7 on the Program Nomination Form. Both items are designed 
to identify reasonable, verifiable proofs of success for nominated programs. 
Evidence may include—but need not be limited to—statistical, "hard" 
evaluation data; however, such evidence must be clearly defined. 

We are particularly interested in knowing if you have statewide test data 
which may help in the identification of schools that have outstanding pro- 
grams. 

Since we are working on a very tight time schedule, we would appreciate a 
response from you at your earliest convenience. 




Cordially yours. 



^^n E. Bowers 
Project Director 
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VERSION 3, Large-City Superintendents 
Professional Organizations 
Other Groups American 

INSTITUTES 




ton RESEARCH 

P.O. Box 1113 • Palo Alt6, California 94302 • (415) 493-3550 • Cable: AlRESEARCH 



The National Right to Read Program of the U. S. Office of Education 
b-is selected the American Institutes for :ReseaTch to identify up to 25 
outstandingly effective reading programs that are now In use In American 
schools and other institutions and to publicize these programs to other 
school systems and institutions that may wish to adopt them- 

You have been selected as a person knowledgeable about innovative and 
exceptionally successful reading programs. We hope that you and your 
staff will be willing to participate in this study by nominating one 
or more reading programs that you know to be successful as candidates 
for inclusion in the group to be publicized and "packaged** for distribu- 
tion. We are specially concerned about the process which will create a 
setting for successful output from this project. "Packaging** in the 
sense we are using it is more broadly conceived than is generally 
interpreted. The **packages** from this study will stress the total com- 
prehensive planning activity necessary to successfully duplicate the 
program elsewhere, but will not include the specific instructional 
materials used by students and teachers in the classroom. Accordingly, 
I am enclosing several Program Nomination Forms. If you should need 
additional copies of the form, 1 shall be glad to send them. 

The program or programs that you nominate may operate at any level from 
preschool through adulthood. A program, for our purposes, is defined, 
In the case of a school district, as one which is used throughout the 
school building at a given grade level, in a cluster of grades, or for 
a special population (such as a program that is being implemented through 
out the primary grades or in all eighth grades or for all deaf children). 
An effort leveled at only one fifth-grade class where there are other 
fifth-grade classes would not qualify as a program. Neither would the 
efforts of a single teacher who is **very successful in teaching reading** 
qualify as a program. Very large units, such as programs being imple- 
' mented statewide, are of interest to us and may be nominated for con- 
sideration. A program, like a package, is conceived of in a global 
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sense and Includes all those things necessary to Implement It success-- 
fully. They may be aimed at special subgroups at any of these age levels. 
We are interested in programs funded by local tax sources and private 
sources, e.g., business, industry, foundations, etc., as well as those 
funded by states and the federal government. 

A list of criteria for selection of outstanding reading programs. Is 
enclosed. If you think a program might meet these criteria but are 
not sure, please nominate it anyway. If you would like to nominate 
a program which does not yet have statistical evidence of success, do 
so. In either case, please give particularly careful thought to your 
answer to Items 6 and 7 on the Program Nomination Form. Both items are 
designed to identify reasonable, verifiable proofs of success for 
nominated programs. Evidence may include — but need not be limited to — 
statistical, '*hard" evaluation data; however, such evidence must be 
clki:^rly defined. 

Since we arc working on a very tight time schedule, we would appreciate 
a response from you at yoi'^ earliest convenience. 




^hn E. Bow^s 
Project Director 
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VERSION 4, Adult/Special Education Contacts 

£^ "^^V^% AMERICAN 

MjL I ■■v. INSTITUTES 

^ ^bA^b^r FOR RESEAf 



P.O. Box 1113 • Palo Alto, California 94302 • (415) 493-3550 • Cable: AIRESEARCH 



The National Right t Read Program of the U. Office of education 
has selected the American Institutes for Research to identify up to 25 
outstandingly effective reading programs that are now in use in American 
schools and other institutions and to publicize these programs to other 
school systems and institutions that may wish to adopt them. 

You have been selected as a person knowledgeable about innovative and 
exceptionally successful reading programs. We hope that you and your 
staff will be willing to pidrtlcipate in this study by nominating one 
or more reading programs IhLt you know to be successful as candidates 
for inclusion in the group to be publicized and ''packaged** for distribu- 
tion. We are specially concerned about the process which will create a 
setting for successful output from this project* **P,ackaging'* in the 
sense we are using it is more broadly conceived than is generally 
interpreted. The ''packages** from this study will stress the total 
comprehensive planning activity necessary to successfully duplicate 
the program elsewhere, but will not include the specific instructional 
materials used by students and teachers in the classroom* Accordingly, 
I am enclosing some Program Nomi^nation Forms. If you should need 
additional copies of the form, 1 shall be glad to send them.. 

A program or programs that you nominate may operate at any level from 
preschool through adulthood. We expect to receive hundreds of nomina- 
tions for state and federally funded school-based programs* We have 
contacted you because we also want nominations for programs which 
serve special subgroups f the population. Specifically, we are 
seeking reading programs for adults, including those for teenagers 
who have dropped out of school, and reading programs for children in 
special education classes. We hope you will consider as nominees pro- 
grams which may be funded by loc.al tax sources and private sources, 
e.g., business, i . 'ustry, foundations, etc., as w^ll as those funded 
by states and the federal government. 
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A list of criteria for selection of outstanding reading programs is 
enclosed • If you tlunk a program might meet th^ise criteria but are 
not sure, please nominate it anyway. If you would like to nominate 
a program which does not yet have statistical evidence of success, do 
so. In either case, please give particularly careful thought to your 
answer to Items 6 and 7 on the Program Nomination Form. Both items are 
designed to identify reasonable, verifiable proofs of success for 
nominated programs. Kvidence may include — but need not be limited to — 
statistical, "hard*' evaluation data; however, such evidence must be 
clearly defined. 

Since wc are working on a vety tight time schedule, we would appreciate 
a response from you at your earliest convenience. 



Cordially yours, 




^)<Hin Bowers 
Project Director 
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PROGRAM NOMINATION PANEL 



Dr. Leon Williamson 
Dr. Mavis Martin 
Nancy B. Davis 
Sister Beverly Bell 
William Durr 
Merri Warren 
John Manninn 
Clarice Millam 
Thomas C. Hatcher 
Vernice Hubbard 
H. Alan Robinson 
David Yarington 
Egon Guba 
Irene J. Athey 
Marie Hackett 
Edward B. Fry 
Dolores Durkin 
John Follman 
Morton Botel 
Helen Huus 
Marjory S. Johnson 
Jerry Walker 
Roy Butz 
Laura Zirbes 
Constance M. McCullough 
William Sheldon 
Theodore L. Harri*; 
Robert Bradfield 
Mrs. Charlotte Brooks 
Dr. Lilyan Hanchey 
Sonnie McCullough 
Joseph Fisher 
Dr. Eric Thurston 
Louella Burmeister 



Dr. Wesley Mieirhenry 
Dr. Asa Hillard 
Terrell Bell 
Dr. Ester J. Svenson 
Russell Jackson 
Albert Harris 
Helen Kyle 
Fred Strodtbeck 
Doris Entwistle 
Fredelle May nurd 
Edwii L. Palmer 
Gerald S. Lesser 
Alton Raygor 
Harry Singer 
Edmund Gordon 
Albert N. Hieronymus 
Warren Finley 
Rev. C. Albert Koob 
Milagros Aquino 
Meyer Weinberg 
Alcnzo Perales 
Nancy St. John 
Richard I. Miller 
Leon M. Lessinger 
Wilbur J. Cohen 
Boyd Mcpandless 
Leon Hall 
Roland Nagle 
Robert Hess 
Edward C. Pino 
Grayce A. Ransom 

, Anne McKillop 

i 

Barbara ..ateman 
William Cruickshank 



Richard L. Carnrr 
Dr. Harvey Goldman 
Walt Wolfram 
Lila Gleitman 
K.S. Goodman 
Albert J. Kingston 
Eleanor J. Gibson 
Jack W. Lombard 
Mrs. June Durand 
Richard L. Smith 
Or. Thomas Fitzgibbon 
Dr. Samuel Messick 
Scholastic Test Service 
Teachers College Press 
Lyons, Carnahan, Inc. 
Mary McNulty 
Hon. Alonza Bell 
Hon. John Dellenback 
Hon. Edwin D. Eshleman 
Sen. Peter H. Dominick 
Sen. Jacob K. Javits 
Sen. Richard S. Schweiker 
Hon. Carl D. Perkins 
Hon. Elliott Hagan 
Hon. Daniel J. Flood 
Hon. Edith Green 
Hon. John Brademas 
Sen. Warren G. Magnuson 
Sen. Claiborne Pell 
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PROFESSIONAL ORGANIZATIONS AND OTHER GROUPS* 

Adult Education Association of the U.S.A. 
Adult Education Regional Programs* 
American Association of Mental Deficiency 

American Association of Publishers (and other publishers/companies) 

American Association of School Administrators 

American Federation of Teachers 

American Library Association 

American Speech and Hearing Association 

ASCD Urban Curriculum Leaders* 

Association for Childhood Education International (now called Cooperative 

Development Program) 

Association for Children with Learning Disabilities 

Association for Supervision and Curriculum Deye^opmnt 

Association of Black Psychologists 

Association of University Evening Colleges 

Council for Basic Education 

Council for Exceptional Children 

Deans of Continuing Education* 

Institute of the Blick World 

International Reading Association 

Joseph P. Kennedy Foundation 

Junior College Organizations* 

Literacy Volunteers of America 

National Affiliation for Literacy Advance 

National Association for Public Continuing and Adult Education 

National Association for Retarded Children 

fational Association of Black Students 

National Association of Elemantary School Principals 

National Council of State Education Associations 

National Council of Teachers of English 

National School Boards Association 

National Society for Autistic Children 

National University Extension Association 

Negro Bibliographic Research Center 
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Regional Right to Read Representatives* 

Special Education Personnel In State Education Agencies* 

State Administrators for Junior Colleges* 

State Association of School Administrators 

United Cerebral Palsy Association 



*Hany of these groups were contacted as part of AIR's Intensified search 
for adult programs, special education programs, and programs operating In 
the community rather than 1n school settings. 
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AMERICAN Institutes for Research 



orf ict of ManatMitnt intf lutfgil 

No. SI • srxM 
Approval Cxplrit 30 Junt I9M 



( >nter for BeHParch and Ri<aluation 

in ihf ApplicationH 0/ Technology in Hduvalion 



PROGRAM INFORMATION FORM 

Identlfyinq, Validating and Mu1t1 -Media Packaging 
of Successful Reading Programs 



A Project sponsored by 
The National Right to Read Program 
U.S. Office of Education 



IMPORTANT NOTICE 
Please answer each Item carefully and completely 
before you return this form. The Information you 
provide w111 be used for evaluating your program for 
further consideration In this study. 
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PROGRAM INFORMATION FORM 



SECTION I Identification Information 



Program Title 



(i; 



Program Director 
Address 



Title 
Phone 



(21 



Sponsor (school district or other) 

Superintendent or director 

Address 



Phone 



(3] 



Address where your proqram Is operating 



(41 



If you know any* please 11st one or two other school districts or sponsorlnn 
Institutions where the total program named In Item 1 Is being duplicated. (S, 



Educational Facility 



Address and Zip Code 
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SECTION II — Program Scretnlna Information 






In this itctlon, all qutstlons art numbered at the right-hand edge; please answer 
each by marking the letter X In the appropriate box. 




Is cognitive improvement in readlhq 
and reading-related skills a major 
focus of your program? 


YES 

□ 


NO 


(6) 




Less than a vear 


□ 


(7) 


fiOM long has your program 
been operating continuously? 


At least a year but less than 
two years 

Two years but less than three 


□ 
□ 




• 


Three years or more 


□ 






None of these 

(If none, Indicate whv) 




Do you plan to keep your program 
operating for at least two more 
years (through the 1974-75 academic 
year)? 


YES 

r— -i 

(If no, Indicate why not)' — ' 


NO 

□ 


(8) 




None available and none 
planned 




(9) 




None available but Initial 
steps taken 


□ 




Are evaluation reports (e.g., 
batftlinA tfttt data, rt-tftst 
data, measures of the program's 
effect) available? 


Available out not published 

Most recent publication 
prior to 1/1/68 

Most recent publication 
since 1/1/68 


□ 
□ 
□ 






Three years or more 


□ 


(10) 




More than two years, less 
than three 


□ 




Evaluation data are available 

for how long? 


More than one year, less 
man lwo 

Only one year 

Less than one year 


nnn 






Not available 
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Art tht data evaluating youi' 
total program approach available 
for one or moi*e sites? 



How many participants or 
Individual records are In- 
cluded In the evaluation? 



More than one site (e.g. • 
more than one school) 

Your site only 

Not available 



Less than 10 
10 to 29 
30 to 49 

50 to 99 
100 to 199 
200 to 499 

SOO or more 



□ 
□ 
□ 

n 

D 



(11) 



(12) 



J** If you marked any one of these boxes, do not complete this form or send program documents* 
[ INSiEAD, please detach the first 3 pages and return them so that your response can be recorded. 



What measures have been 
analyzed to show the success 
of your program? 



Analysis of nationally 
standardized reading 
test results 


YtS 

□ 


NO 

□ 


(13) 


Analysis of locally 
developed reading test 
results 


□ 


□ 


04) 


Analysis of nationally 
standardized general 
abl 1 Ity measures 


□ 


□ 


(IS). 


Analysis df locally 
developed general 
ability measures 


□ 


□ 


(10 


Analysis of other program 
success Indicators (e.g.^P"] 
observations* affective ^ — ' 
measures 1 teacher records » 
questionnaires) 




(17) 


Other procedures 
(Please specify) 


□ 


□ 


(11) 



Mtasures not yet i — i i — i 

inalyzed [J [J (19) 
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Art up-to-date program descriptions available (e.g., 
staff, participants, schedules and activities)? 



YES NO 

□ □ (20) 



What kind of Improvement or gain 
by program students was found? 



The mean test score of 
the students exceeds a 
specified norm 

A mean gain over 
exactly one year Is 
bigger than expected 

A mean gain for less 
than one year Is 
bigger than expected 

The mean of students in 
the program exceeds 
that of comparable stu- 
dents not In the pro- 
gram 

The mean gain of stu- 
dents In, the program 
Is greater than for 
comparable students 
not In the program 

Some other Improvement, 
not one of these 
(Please specify) 



YES 

□ 



NO 
□ 



□ □ 

□ □ 



□ □ 



□ □ 

□ □ 



(21) 



(22) 



(23) 



(24) 



(25) 



(26) 



How significant were the statis- 
tical results showing the effect 
of your program? 
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No tests of significance 
were made 

No significant differences 
found yet 

The program showed differences 
significant between the 5 and 
10 percent one-tailed (10 to 
20 two- tailed) level 

The program showed differences 
significant at better than the 
5 percent one- tailed (10 per 
cent two- tailed) level 

The program showed differences 
significant at bettei^ than the 
one percent one- tailed (two per 
cent two- tailed) level' 



□ 
□ 

□ 

□ 
□ 



(27) 
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By what amount dots the annual 
ptr-pupll cost of this program 
txettd that of the regular 
district program? 



Less than $50 
$50 to $99 
$100 to $199 
$200 to $299 
$300 to $399 
$400 to $499 
$500 to $999 
$1000 or more 



□ 
□ 
□ 
□ 
□ 
□ 
□ 

□ 



For what target population of 
students Is. your program designed? 



Unselected cross section 

Mentally retarded 

Bilingual 

Disadvantaged 

Physically handicapped 
(deaf, blind,' etc.) 

Institutionalized 

Other groups 

(Please specify below) 



a 




(2») 


□ 


□ 


(30) 


□ 


□ . 


(31) 


□ 


□ 


(32) 


□ 


□ 


(33) 


□ 


□ 


(34) 


□ 


□ 


(3S) 



Are 20 percent or more of your 
proaram students In any of the 
following categories? 







NO 


(36) 


American Eskimo, Aleut, 
or Indian 


□ 


□ 


Black 


□ 


□ 


(37) 


Oriental or Asian 


□ 


□ 


(3a) 


Spanish-speaking 


□ 


□ 


(39) 


White 


□ 


□ 


(40) 


All others 


□ 


□ 


(41) 
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In which area do the majority of 
the program students live? 



Rural and small town 
of less than 10,000 

Small city of 10«000-199,000 

Small city suburbs 

Inner area In large city 
of 200,000 or more 



□ 

□ 
□ 

□ 

Residential area in large city 
Suburbs of a large city 



(42) 



What Is the average family Income 
level of students In the program? 



Are specific diagnostic tech- 
niques or instruments used to: 



Low Income (under $6,000) Q 
Middle Income ($6. 000- $15, 000) 
High Income (above $15,000) 



Determine each student's 

level of reading readi- YES NO 

ness or skill (e.g., his I 1 I I 

reading grade level)? ' — ' • — ' 
(If yes, please specify) 



Determine each student's YES NO 
strengths, weaknesses, FH 11 
and difficulties In Ian- ' — • • — ' 
Quage and reading skills 
(e.g., difficulty with 
decoding)? 

(If yes, please specify) 



(43) 



(44) 



(45) 



ERIC 



In this i^rogram, how many hours 
per week are scheduled for the 
subject Language Arts? 



lOS 



Less than 2 hours 
2 hours to 3 hours 59 minutes 
4 hours to 5 hours 59 minutes 
6 hours to 7 hours 59 minutes 
8 hours to 9 hours 59 minutes 
10 hours to 11 hours 59 minutes 

12 hours or more 



□ 
□ 
□ 
□ 
□ 
□ 



(4<) 
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What summary statistics were 
used In the analysis of program 
data? 



Means or medians 

Standard deviations 
or variances 

Cover lances or cor- 
relation coefficients 

Frequency counts, per- 
centages, or propor- 
tions 

Significance tests 

Methods not mentioned 
above 

(Please specify) 



J8L, 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 

□ □ 



How large was the estimated 
program effect on achievement 
(I.e., the average gain of 
students In the program over 
and above the gain expected 
In a comparison group)? 
(If more than one estimation, 
give the higher figure only.) 



One-tenth of a standard 
deviation unit 




n 
1 1 


One-fifth of a standard 
deviation unit 




□ 


One-quarter of a standard 
deviation unit 


□ 


One- third of a standard 
deviation unit 




□ 


One-half of a standard 
deviation unit 




□ 


Better than a half SD 




□ 


Gain cannot be given this way 


□ 


Age 


YES 

□ 


NU 

□ 


Sex 


□ 


□ 


Grade level 


□ 


n 


Ethnic proportions 
In group 


□ 


□ 



Which of these factors were taken 
Into explicit account In the 
analyses of BOTH program AND 
comparison data? 
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What was the reliability co- 
efficient of the test used 
to measure reading achieve- 
ment for this program? 



Between .6 and .69 

Between .7 and .79 

Between .8 and .89 

.9 and over 

Given by publishers 
for standardization 
group only as over .8 

Not yet determined 
No such test was used 



□ 
□ 
□ 
□ 

□ 
□ 
□ 



Whdt percentage of annual 
attrition or loss of students 
from the program was allowed 
for, to correct for bias In 
statistical analysis (e.g., by 
eliminating froiii consideration 
persons who start the program 
but do not finish)? 



Was 15 percent or more 

Was between 10 and 14.9 
percent 

Was between 5 and 9.9 
percent 

Was oetween 0 and 4.9 
percent 

No allowance was made 
for losses 

No losses occurred 



□ 

□ 

□ 

□ 

□ 
□ 



How similar were the 
pre- and post-tests used to 
determine gain in reading 
Skins? 



Were identical 

Were parallel forms of a 
single test 

Were consecutive forms from 
the same source 

Were similar in form, but 
from different sources 

Only one test has been 
applied 

Tests were not of readinq 
skills 

No tests were applied 



□ 

□ 

□ 

□ 

□ 

□ 
□ 
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SECTION III - Brief Descriptive Information 

In what year did the program begin operation? ____ 

(Year) 



How many program participants and classes (groups) are there In your program? ($\ 
If this Is a school -based program. Indicate enrollment by class and grade level. 

Grade or Number of Number of Classes 
Other Level Participants or Groups 



Please list the major Instructional strategies used to help clients Improve (6; 
their reading -re la ted ski 1 is. Briefly Jescribd, If necessary. 



Are there any major program features (e.g., parent involvement) which are (ft 
not Included in your I'st of instructional strategies above and are not 
Included in your list uf key program objectives in the chart at the end of 
this questionnaire? If so, please list up to three (3) of these major pro- 
gram features. Briefly describe, if necessary. 




U6 
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Pluse lin up to five (5) klnds of materials and/or equlpinent which are 
absolutely Indispensable for your programi noting their availability as 
"commercial," "district," "teacher-prepared," "student-prepared," or other 
appropriate comment. 


(6S) 


Most Essential Items of Materials Quantity for 

and/or Equipment 30 Students AvallabllUy 




Where are program activities physically located? If any special features 
were provided to suit these facilities to the program, briefly note. 


(66) 


Location of Program Activities Special Features 




What Is the total cost of Instructional materials for a class of 30? 


(67) 


$ for a class of 30, to the nearest dollar 




To the nearest dollar, roughly what portion of the funds currently required 
to maintain the program come from the following sources? Please specify 
the exact source for each category, e.g., Title III. 




1 Federal 




f State 




^ Local 




^ Private 




What it the avftrage. annual per-DUoil cost for the district's reauiar ichooi 
program (I.e., cost per pupil for students outside the special reading pro- 
gram described herein)? 




$ , per-pupll cost for regular program 
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Return to: 
John E. Bowers 

American Institutes for Research 
P. 0. Box 1113 

Palo Alto, California 94302 



We would appreciate It If you 
would return this form by 
First Class mall. Documents 
may be sent under separate 
cover. 
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VERSION 1. School Programs 

^PV^ AMIRICAM 
I mmWm INITITUTIt 
^^BJhKw ran RINAHCH 

P.O. Box 1113 t Palo Alto, California MSOa • (418)403-3SSO • CablO: AIRE8IARCH 



The National Rl^ht to Ruad Program of thc.U. S. Office of Education 
han selected the Amarican InBtltutes for Reneiirch to identify up to 25 
outstandingly effective reading programs that arc now In use In American 
schools and other institutions and to publici^fc* thoso programs to other 
school systems and Institutions that may wish to adopt them. 

Your district has been nominated as one which may qualify as having 
a reading program among the best in the nation. I7c hope you will be 
willing to provide Information on your program so that we may consider 
it along with other nominees. 

Information about the 25 programs selected for special publicizing 
will be packaged and made available to describe the exemplary models to 
other school districts. In addition, many more programs will be described 
in a nationally disseminated catalog of reading programs, tf your program 
is selected, full credit and acknowledgment will be given to you and your 
district in the catalog or package. Packaging in the sense we are using 
it Is more broadly conceived than is generally the case. The packages 
produced in this study will stress the plans and activities necessary to 
duplicate the program successfully « and not Just the specific Instructional 
materials used by students and teachers in the classroom. Similarly, a 
program will constitute the total set of conditions we wish to identify 
and package. 

As is implied by these broad definitions, part of our task is to 
collect detailed information cn programs being considered for this study. 
For the purpose of collecting initial information about your program, we 
have enclosed a Program Information Form. We have also enclosed a list 
criteria so that you may judge whether your program qualifies as a candidate 
for our study. If you feel your program meets these criteria, we ask your 
cooperation in carefully and accurately completing this form. We realize 
that the form is long and will require some expenditure of effort on your 
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part. However, the poit»ntial payoff for you and your district is also 
considerable if your program should be selected. 

Many of the items on the Program Information Form deal with statis- 
tical and evaluative techniques which may not havo becMi used in your 
program. Please do not conclude that your program must meet overly severe 
standards of evaluation rli;or in order to qualify for this study. We 
realize the practical limits within which real programs must operito. Wo 
will therefore be cataloging and packaging a wldi^ range of reading programs. 
If you feel that your pro g ram has special merit » we e ncourage you to com- ' 
pl ete the for m. All programs returning completed forms will receive 
careful consideration for cataloging and packngln?',. 

Wc wish to base our tivaluation of your program on as much informatltm 
as possible. Wlillt! the l^rogram Information Form does give us a good 
starting point » we also wish to examine whatever written documents you 
have describing your program. We are especially Interested in seeing the 
results of any evaluation you have made. For tills reason* we ask that 
you send us* as complete a set of written documents as possible when you 
return the Program Information Form. 

In addition to programs in schools, we are Interested in finding 
outstanding programs which may be functioning In or near your community 
through other agencies, if you know of any such programs, we would 
appreciate knowing about them and whom to contact. A Program Nomination 
Form is enclosed for this purpose. 

There is some urgency in receiving a prompt reply from you since uc arc 
scheduled to begin packaging an early date. In order for a program to be 
considered, the Program Information Form together with all printed reports 
about your program must be received by us no later than 15 February or 
sooner if possible. 



Should you have questions, please call me or Mrs. Peggie Campeau at 
(415) 493-3550 or at (408) 354-9088. 



P. S. As you complete the Program Information Form, please note that in 
a YES/NO item, either YES or NO should be marked for every pair of 
boxes in the numbered response column. 




Sincerely, 



John £. Bowers 
Project Director 
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VERSION 2. Superintendents of Schools 




AMmiCAM 
INtTITUTIt 
POR RISIARCH 



P O Box 1113 • Palo Alto, California 94302 • (416)4W3^580 t Cablo: AIRESEARCH 



The National Right to Read Program of the U. S. Office of Education 
hai selected the American Institutes for Research to Identify up to 25 
outstandingly effective reading programs now In use In American schools 
and other Institutions. In-depth Information packages on the programs 
selected will be made available to other educators as exemplary modeli 
with full credit and acknowledgement to program staff and school district. 

In your district an outstanding reading program has been nominated 
which may qualify for our study. Our time schedule requires very rapid 
Identification of those exemplary programs for which we will be compiling 
information packages. Therefore, we have taken the liberty of contacting 
the staff of the above program. They should receive a letter and a 
questionnaire, the "Program Information Form," about the name time this 
letter reaches you. Wc would of course be most grateful for any action 
you or your staff may wish to take that will expedite the return of the 
Information we need. 

Thank you for your cooperation. 
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VERSION 3. Non^school Prograni 

■ INtTITUTM 
g^^^JLKw PORIIINAI 



P.O. Box 1113 • Palo Alto, California 94)08 • (418)493^8S0 • Cablo: AIRE8EARCH 



The National Right to Read Program of the U. S. Office of Education 
has selected the American Institutes for Research to Identify up to 25 
outstandingly effective reading programs that are now in use In American 
schools and other Institutions and to publicize these programs to other 
school systems and Institutions that may wish to adopt them. 

Your agency/lnstltutlon has been nominated as one which may qualify 
as having a reading program among the best In the nation. We Hope you 
will be willing to provide Information on your program so that we may 
consider it along with other nominees. 

Information about the 25 programs selected for special publicising 
will be packaged and made available to describe the exemplary models to 
other agencies/institutions* In addition, many more programs will be 
described in a nationally disseminated catalog of reading programs. If 
your program is selected, full credit and acknowledgment will be given to 
you and your agency/institution in the catalog or package. Packaging in 
the sense we are using it is more broadly conceived than is generally the 
case. The packages produced in this study will stress the plans and 
activities necessary to duplicate the program successfully! and not just 
the specific Instructional materials used by students and teachers in the 
classroom. Similarly, a program will constitute the total set of conditions 
we wish to Identify and package. 

As is Implied by these broad definitions, part of our task is to 
collect detailed Information on programs being considered for this study. 
For the purpose of collecting Initial information about your program, we 
have enclosed a Program Information Form. We have also enclosed a Hat 
of criteria so that you may Judge whether your program qualifies as a 
candidate for our study. If you feel your program meets these criteria, 
we ask your cooperation in carefully and accurately completing this form. 
We realize that the form is long and will require some expenditure of 



1 6 An Iqual Opportunity Imployor 
D-18 



effort on your part. However, the potential payoff for you and your agency/ 
Institution Is al^o considerable if your pro^rani should be selected. 



Many of the Items on the Program Information Form deal with atatls* 
tlcal and evaluative techniques which may not have been used In your 
program. Please do not conclude that your program must meet overly severe 
standards of evaluation rigor In order to qualify for this study. We 
realize the practical limits within which real pro;?rams must operate. 
We will therefore be ntaloging and packaging a w^Je range of reading 
programs. If you tool that your program has special merit, we enc ourage 
you to complete the t orm. All programs returning completed forms will 
receive careful consideration for cataloging and packaging* 

We wish to base our evaluation of your program on as much Information 
as possible. While the Program Information Form does give us a good 
starting point, we also wish to examine whatever written documents you 
have describinp your program. We are especially interested in seeing the 
results of any evaluation you have made. For this reason* we ask that 
you. send us as complete a set of written documents as possible when you 
return the Program Information Form. 

We are especially Interested In finding other outstanding programs 
which may be functioning In or near your community through other agencies. 
If you know of any such programs, we would appreciate knowing about them 
and whom to. contact. 

There is some urRcncv in receiving a prompt replv from you since we are 
scheduled to begin packaging at an early date. In order for a program to be 
considered, the Program Information Form together with all printed reports 
about your program must be received by us no later than 
sooner If possible* 

Should you have questions, please call me or Mrs. Peggie Campeau at 
(415) 493-3550 or at (408) 354-9088. 




Sincerely yours, 



P. S. As you complete the Program Information Form, please note that in a 
YES/NO Item, either YES or NO should be marked for every pair of 
boxes in the numbered response column. 
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APPENDIX E 



Inventory of Program Data 

Outline for the Instructional Handbook 

Outline for the Program Management Handbook 
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PROGRAM CODE NO. 



INVENTORY OF PROGRAM DATA (IPD) 



VALIOATOR(S): . 
WRITER(S) : , 
SITE VISITORS: 



AS A RESULT OF OUR IN-HOUSE VALIDATION. THIS 
PROGRAM IS RECOMMENDED FOR (check one box): 



CATALOG only- 



PACKAGE 



□ 




HIGH priority—. 
-MODERATE priority- Q 
IGH priority--- Q 
MODERATE priority- Q 



1.00 IDENTIFICATION 



XEROX AND AFFIX THAT PORTION OF PAGE 1 OF THE PIF 
WHICH SUPPLIES THE FOLLOWING IDENTIFICATION INFOR- 
MATION: 

PIF Item 1 : Program Title 

PIF Item 2: Program Director's Name, Title, Address, 
and Phone 

PIF Item 3: Sponsor, Superintendent or Other Direc- 
tor's Name, Address, and Phone 

PIF Item 4: Address Where Program Operates 

PIF Item 5: Other Sites 



ERIC 



E-l 



119 



INVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 



Shidtd right margin Indlcttti that the qutstlon should be 
Ignorad for CATALOG programs and answtrad for PACKAGE programs. 



1.01 For further Information: 

For CATALOG program, name and address which should appear 
In the "For Further Information" section (Note below.) 




□ 



For PACKAGE program, name and address of person who 
coordinates visits (Note below.) 



Complete the Information below for each person contacted In the course of 
program validation or preparation of the PDF or INFOPAK. 




Name and Title 



Address 



Phone 



"PHy contact? 



Person who filled 
out PIF 



Person who should ap- 
pear In the "for fur- 
ther Info" section. 
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INVENTORY OF PROGRAM DATA (IPO) PROGRAM CODE N0._ 

1.02 Program documentation: (Also 11st AV documentation. If any, and Its cost 

and availability.) 



Docunent Number 


Hibilographic Reference 


PIF 


(Program Information Form) 


1 


(Notes from phone contacts, document review, site visits.) 


2 

(etc.) 
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INVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 





DOC. NO 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


Need 

MORE 
INFO 


2.00 PARTICIPANTS 








2.01 Age/grade and special characteristics on PIF 
agree with documents 


PIF 


29-35, 
62 


□ 


2.02 Selection procedures 

(dlaonostlc tssts. oarsnt reouest admlccinn 

panel, referral system, etc.) 






□ 


2.03 Selection criteria 

(qualifications, background, skills, etc.) 






□ 


r 3.00 PROGRAM SIZE 








3.01 Program enrollment data on PIF agree with 
documents 


PIF 


62 


□ 


4.00 OBJECTIVES 








4.01 Ke^ objectives on PIF agree with documents 


PIF 


63,64 

71 


□ 
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VENTORY OF PROGRAM DATA (IPD) PROGRAM CODE NO. 





uuu. NO 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


NEED 
MORE 
INFO 


S.OO LOCALE (DEMOGRAPHY) 








5.01 Demographic background of participants on 
PIF agrees with documents 


PIF 


36-43 


□ 


5.02 Population characteristics of locale which 
affect program: 

Transiency 






r~| 
1 1 


Second language or strong dialect Influence^ 






1 1 


Other 






□ 


5.03 (Question for PACKAGE program operating 
IN SCHOOL:) 

Characteristics of school district 
which affect program: 

Size 




ie»«ai«e«i«< 


• 

• 

n 

hmmmmmmmmm 


District policies 






□ 
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NVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 





M, NO. 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


NEED 
MORE 
INFO 


6.00 PROGRAM HISTORY 








6.01 PIF and documents agree with respect to year 
program started 


PIF 61 




□ 


6.02 Initial stimulus for program: 

Effort of key person (get contact Info) 






□ 


Outgrowth of earlier effort (Any modifications?) 






□ 


Other 






□ 


6.03 Needs assessment done? If YES: 
By whom 






■■■■Pi 


Methods/ Instruments 






□■ 


Specific needs Identified 






iU! Uj 

k m a 


Needs selected for program (priorities) 






Who makes these decisions? 






□ 


Useful materials for package 
(e.g., questionnaires) 






□ 



ERIC 
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VEHTORY OF PROGRAM DATA (IPD) PROGRAM CODE NO. 





DOC. NO. 
See key, 
IPD p. 3 


OR 
REF. 




NEgD 
MORE 
INFO 




6.04 


Planning process (before program began): 
Major steps 






n 


Key ptrsons/roles In major planning tasks 






□ 


Useful materials for package (e.g., flow charts) 






□ 


6.05 


Implementation process: 
Major steps 




• 




□ 




Key persons/roles In major Implementation tasks 








□ 




Effective/ Ineffective procedures 








□ 




Useful materials for package (e.g., schedules) 








□ 




6.06 


If program has changed from original plan: 
How? 

(changes In goals, activities, staff, etc.) 








□ 




Why? 

(recruitment problems, community pressure, 
budget cuts) 






□ 




6.07 


Program's growth since It began 






d 
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NVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 





DOC. NO. 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


DIUO 
LaI o Z 




7,00 STAFF REQUIREMENTS 








7.01 Staff requirements on PIF agree with documents 
(highlight essential qualifications; double- 
check for consultants and volunteers) 


PIF 


70 


□ 


7.02 Inservlce training (additional details): 
Specific objectives 






1 


□ 

••••• 

□ 


Key persons/roles In planning/conducting 






1 


When and where (get schedule) 






□ 


Instructional methods/activities 






1 


WW 

c 


y 


Assessment of staff proficiency In skills taught 






□ 


7.03 Problems In recruiting or maintaining staff: 
Incentives to attract staff 








Teachers allowed to leave program 






□ 

□ 


Other 



ERIC 
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IVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 





uQC. NO 
See key. 

Ton n 9 

IPD p. d 


PAGE 
OR 

DTP 


NEED 
MORE 
INFO 


8.00 PROGRAM MANAGEMENT 








8*01 Chain of command* 

Protocol for who te11s whom what (get organization 
chart) 






□ 


Day-to-day situations In which above routine 
Is modified to facilitate program management 






□ 


Procedures for decision-making re unsolicited 
advice from teachers, students, and parents 






□ 


8.02 Periodic review of program (process evaluation): 
Proaram orocedurcs 4«kSessAd 

r 1 W^l will W WWM 1 Ww • 






□ 


Who doss what? (Ic§v otrsons/rolAsI 






MM 

□ 


Method/ Instruments used 






□ 


Measurement schedule/frequency (get schedule of 

rhAckool t /of h^r r#rAvtHc1 








riojui uvwiaiuiia inouv uii Dfl9l9 UT review 






□ 

□ 


Who makes these decisions 






o.u<9 upinions niosb sssenbisi niansgenisnb bvcnnic)ues 
for repHcdtlna program (Indicate Identity of 
person expressing opinion) 
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INVENTORY OF PROGRAM DATA (IPD) 


PROGRAM CODE NO. 




DOC. NO 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


NEED 
MORE 
INFO 


9.00 THE INSTRUCTIONAL PROGRAM 








9.01 Strategies and features on PIF agree 
with documents 


PIF 


63, 64 


□ 


9.02 Instructional activities (more detail): 
Teaching techniques (specific examples) 






□ 


Classroom management/motivation .techniques 






□ 


Feedback techniques 






□ 


9.03 Grouping patterns used: 

Purpose of each type of grouping 






□ 


How frequently used 






□ 


Describe an event for each grouping 








9.04 Typical schedule of program activities (Indicate 
oerlod. davi week, other unit of time covered by 
schedule; minutes or portion of time per activity; 
whether schedule rotates.) 






□ 


Obtain Or produce a time line which shows the 
sequence of program activities for a typical 
oartlclpant from the time he enters the program 
until he completes the program. Also Indicate 
the range of weeks per activity which might be 
required due to the range of participants' 
abilities. 






□ 



er|c 
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NVENTORY QF PROGRAM DATA (IRDJ 


PROGRAM CODE NO. 




m, NO 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


NEED 
MORE 
INFO 


9.05 On-going diagnosis and assessment of participants: 
Specific reading-related ski lis/ problems measured 






□ 


Who does what (key persons/roles) 






□ 


Method/ instruments used to measure each 
skill/problem (See also PIF 44, 45) 






□ 


Measurement schedule/frequency 






□ 


Obtain sample copy of individual's diagnostic 
record or profile 






□ 


Decisions based on measurement: 

Prescribe individualized instructional program 






□ 


Group students 






1 — 1 

□ 


Release from program 






□ 


Other 




• 


□ 


9.06 Special provision for teacher planning (regular 
meetings to plan each individual's program, etc.?) 






□ 


9.07 Provision for teachers to modify features 
of the program 






□ 


0 ^ E-11 
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NVENTORY OF PROGRAM DATA PROGRAM CODE NO. 



• 


DOC. NO. 
See key, 
IPD p. 3 


OR 
REF. 


NEED 

.MORE 

'info 


9.08 Parent/conmunlty Involvement: 

wpcw 1 1 iw no vurc ut i nvo i vcincn v 






□ 


Concrete results 






□ 


Measures taken to secure their support 


' 





» M M M M 

LJ 


10.00 FACILITIES 








10.01 Description of program facilities on PIF 
agrees with documents 


PIF 


66 


□ 


10.02 Special construction/alterations required 
for program facilities 






□ 


11.00 EQUIPMENT AND MATERIALS 








11.01 Equipment and materials data on PIF 
agree with documents 


PIF 


65 


□ 


11.02 Purpose/ procedures for use of each 
key Item 

(Relate to specific Instructional events/ 
objectives.) 






□ 



er|c 
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NVENTORY OF PROGRAM DATA (IPD) PROGRAM CODE NO. 





DOC. NO. 
See key, 
IPD p. 3 


PAGE 
OR 
REF. 


NEED 
MORE 
INFO 


12.00 COST 








12.01 Materials cost and regular district cost 
data is complete on PIF 


PIF 


67,69 


□ 


12.02 Compute the percent of program funds 
obtained from each source indicated in 
PIF 68. Enter percents on PIF next to 
dollar amounts. 


PTP 

r i r 


uo 


1 — 1 
U 



(SEE NEXT PAGE FOR 12.03) 
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INVENTORY OF PROGRAM DATA (IPO) 



PROGRAM CODE NO. 



NOTE: TMi cNurt It U bt ^ompltM f«f PACmC program only. Uritt dirtctly on tht 
fon« If MCtmryt Miplttt Ullti for riCh gridt Itvtl or othtr group to bt 
IwclydoTln ttit pocktgid Infonmtlon. (Xorox oddltlonal ch«rti» If ntodtd.) 



12.03 START-UP COST: Include every type of cost paid by the program to Initiate It and carry 

It through Its first year of operation (or other appropriate period). 



Total START-UP cost paid for by program : $ 
Number of participants Included In figure: 
Period of time covered by figure: _ 
Grade or group: 



Per-pupll cost 
paid by program 



(e.g., academic year, '70- '71 J 



Directions: Circle either 1, 2, 3, or 4 for each row of boxes , using the code below. 

1. The program USED Items In this category AND PAID ALL COSTS. 

2. The program USED items In this category BUT PAID ONLY PARTIAL COSTS. 

3. The program USED Items In this category BUT DID NOT PAY ANY OF THESE COSTS. 

4. The program DID NOT USE Items In this category. 



Standard Budget Categories: p^s^Jf" 



Other Budget Categories* ** 



ADMINISTRATION/SUPERVISION 



(salaries, contracted services) 


CD B 


1 IS 


B 


INSTRUCTION: 








Salaries 


a E 


i 01 


B 


Books and Materials 


0 8 


B IS 


B 


Equipment 


0 g 


1 (S 


B 


ATTENDANCE/HEALTH SERVICES: 








Attendance Services 




B IS 


B 


Health Serlvces 


0 B 


B IS 


B 


PUPIL TRANSPORTATION SERVICES: 
(mostly salaries, contracted 

services. Insurance, vehicle 

replacement) 


IS B 


B IS 


B 


OPERATION OF PLANT 
(mostly salaries, contracted 
services, utilities, etc.) 


0 B 


B IS 


B 


MAINTENANCE OF PLANT 
(mostly salaries, services, 
replacement of equipment) 


0 BB IS 


B 


FIXED CHAI^GES 

(employee benefits, rental of 
land and buildings, etc.) 


D B 


B IS 


B 



Q] Q QI B 



If it Is necessary to use Other Budget Categories , show category In CAPS and examples of 
Items Included In the category in parentnese:. 

J 2 (12.03 IS CONTINUED ON THE NEXT PAGE) 



INVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 



12.03 START-UP costs, continued: 



For categories for which you checked Columns 2 or 3 on the chart, please list major 
Items for which you paid only part of the cost or none of the cost. 



If Inservlce training was not covered by the categories shown In the chart, but was 
required for the program, who paid for It? 



Items which were one-time start-up costs: (E.g., building remodeling.) 



Other comments on Interpreting start-up cost data supplied by the program: 



(SEE NEXT PAGE FOR 12.04) 
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INVENTORY OF PROGRAM DATA (IPG) 



PROGRAM CODE NO. 



NOTCi This chart It to bt conplttMl for Ptnom pr^vnm only, mil dlfKtly en Hit 
foni. H ntetsstry» complttt ttbitt for oieH gradt Itvtl or «tlif froup to It 
IneludiTln tht packagod infomatlon. (Xtrox additional chartii If noidad.) 



12.04 CONTINUATION COST: Include every type of recurring cost paid by the program after Its 

first year of operation (or other appropriate period used 1n 12.03). 



Total CONTINUATION cost paid for by program: 
Numbtr of participants Included In figure : 
Period of time covered by figure: 
Grade or group: 



$ 



$ 



Per-pup11 cost 
paid by program 



(e.g., '7Z-'73 academic year) 



Directions: Circle either 1, 2, 3, or 4 for each row of boxes , using the code below. 

1. The program USES Items In this category AND PAYS ALL COSTS. 

2. The program USES Items In this category BUT PAYS ONLY PARTIAL COSTS. 

3. The program USES Items In this category BUT DOES NOT PAY ANY OF THESE COSTS. 

4. The program DOES NOT USE Items In this category. 



Standard Budget Categories: "055^^"! 



ADMINISTRATION/SUPERVISION 
(salaries, contracted services) 


IS B Bl IS 


ffl ■ B B 


INSTRUCTION: 








Salaries 


m B 


D CD IB 




Books and Materials 


06 


SUB 




Equipment 


0 B 


B a 0 




AHENDANCE/HEALTH SERVICES: 








Attendance Services 


OD B 


D QD S 




Health Services 


D B 


DBS 




PUPIL TRANSPORTATION SERVICES: 
(mostly salaries, contracted 

strvlces. Insurance, vehicle 

replacement) 


D e 


0 B B 




OPERATION OF PLANT 
(mostly salaries, contracted 
services, utilities, etc.) 


O Q 


D B B 




MAINTENANCE OF PLANT 
(mostly salaries, services, 
replacement of equipment) 


(XI Q 


D B B 




FIXED CHARGES 

(amployee benefits, rental of 
land and buildings, etc.) 


0 ■ B) El 





Other Budget Categories* ** 



♦**If It If necessary to use Other Budget Categories , show category in CAPS and examples of 
Jl tarns Included In the category in parentheses. 

13*1 E-16 (12.04 IS CONTINUED ON THE NEXT PAGE) 



INVENTORY OF PROGRAM DATA (IPD) 

12.04 CONTINUATION costs, continued: 



PROGRAM CODE NO. 



For categories for which you checked Columns 2 or 3 on the chart, please list major 
Items for which you pay only part of the cost or none of the cost. 



If Inservlce training is not covered by the categories shown In the chart but is 
required for continuation of the program, who pays for It? 



What was the replacement cost for a class of 30 for consumable Items Qf material and 
equipment that must be replaced each year? 



For materials and equipment that were used for more than one year, what was tne re 
placement cost of these reusable Items for ^ class of 30 at the end of: 

2 years? 



3 years? 

4 years? 

5 years? 



Other comments on Interpreting continuation cost data supplied by the program: 



ERIC 



INVENTORY OF PROGRAM DATA (IPD) 



PROGRAM CODE NO. 





DOC. NO. 


paGe 




NEEP 




See key, 


OR 




MORE 




IPD p. 3 


REF. 




INFO 


13.00 MODIFICATIONS PLANNED FOR FUTURf 








13.01 Are modifications In the program planned 
for the future? 






□ 




139 
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OUTLINE FOR THE 
INSTRUCTIONAL HANDBOOK 



Audience: This handbook Is prepared mainly for those who carry out the Instruc- 
tional component of the program. Teachers, and to a lesser degree, 
principals, supervisors, and coordinators of reading Instruction would 
be Included In this group. 

Format ; Headings In ALL CAPS below should be used In the handbook If at all 
possible. Use subheadings which best organize the materials you pre- 
sent. REFER TO HANDBOOKS FOR ALPHAPHQNICS AND FOR THE ALL DAY KINDER - 
GARTEN for additional format Ideas until more handbooks are written. 

IPD Ref ; IPD references are given to shortcut your search for relevant material. 
In addition, the PDF WRITEUP and the OE PRESENTATION will be very 
valuable references, since they Incorporate much of the material In- 
dexed on the IPD. 

IPD REFERENC E 

I. PROGRAM OVERVIEW 



Inttructlons , , 

This section can be written almost entirely from the PDF 
sections which are based on the IPD Items noted below. 
Keep the overview to 300-400 words, or no more than 2 1/2 
pages of letter gothic (large) type, double spaced. Do 
not use subheadings. You do not have to follow this 
sequence, and you can combine points any way you wish. 



o 

ERIC 



A. Identification (very briefly) 

1. Year started 

2. Location name 

3. Program title 

4. Program area (redding, reading •readiness, job 
litfuacy, etc.) 

B. Objectives, main program features/instructionul 
strategies ' 

C. Participant* 

1. Age, grade, demographic background, special 
characteristics 

2. Selection procedures (test, request, referral, 
admission panel) 

3. Selection criteria (qualifications, background, 
sknis) 

4. Number of participants 

D. Locale 

1. Population characteristics of locale which 
affec t program 

2. Characteristics of school district or insti- 
tution/agency which affect nrogram 

E.19 137 



IPD 1.00. 6.01 



IPD 4.00, 9.01 

IPD 2.00, 3 01 , 5.01 



IPD 5.02, 5.03 



IPD REFERENCE 



E. Methodology (may be combined with B above) IPD 9.00 

F. Personnel IPD 7,00 

G. Evaluation Results (very brief and nontechnical) . . (VALIDATOR) 



II. STAFF REQUIREMENTS 



Instructions 

Do not reproduce the PDF chart because It Is In the Program 
Management Handbook . Instead, summarize this Information 
1n snort paragraphs, one for each staff category. If In- 
service training was provided, summarize It without going 
Into the detail provided In the Program Management Hand- 
book . Refer the reader who wants more detail to tnat 
source. 



A. 



B. 



Project staff by category: summarize type, number, 
time devoted to program, qualifications/experience, 
actlvltles/Jutles 



Inservlce: summarize type and extent of Inservlce 
training (objectives, training methods/activities) 



IPD 7.01 

IPD 7.01, 7.02 



III. TYPICAL SCHEDULE/TIME LINE 



IPD 9.04 



Instructions 

Decide on some telegraphic presentation of 
activities ar<2 sequenced. You may want to 
from the PDF. You may want to develop the 
line which could also be used for the Flow 
t Id pant's progress from program entry to 
package components). Accompany the graphi 
with a descriptive, brief narrative which 
Refer to the Instructional Handbook for Al 
an example of presenting and explaining a 



how program 
use the schedule 
sort of time 
Chart of a par- 
exit (one of the 
c presentation 
explains It. 
phaphonlcs for 
day's schedule. 



< 
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PROGRAM ACTIVITIES 



IPD REFERENCE 



Instructions 

Although this section will vary widely for different pro- 
grams, use this general approach: Describe activities 
generally, then zero In to very specific examples. In 
Alphaphonlcs . this section begins with a general summary 
Of activltiei for a day and a week; a sample lesson plan 
and worksheet are reproduced (Plates 1 and 2). Then a 
more detailed description of activities Is presented for 
each day of the week In turn. The last part of the sec- 
tion describes other features of the Instructional pro- 
9^^^' In All Day Kindergarten , most paragraphs or sub- 
sections Include a general description and specific 
examples. However you organize this section, cover the 
following points: 



A. Instructional activities IPD 9.02 

1. Teaching techniques (specific examples) 

2. Classroom management/motivation techniques 

3. Feedback techniques 

B. Grouping patterns used IPD 9.03 

1. Purpose of each type of grouping 

2. How frequently used 

3. Description of an event for each grouping 

C. On-going diagnosis and assessment of participants . IPD 9.05 

1. Specific reading-related skills/problems measured 

2. Who does what (key persons/roles In diagnosis) 

3. Method/ Instruments used to measure each ski 11 /problem 

4. Measurement schedule/frequency 

5. Sample copy of Individual's diagnostic record or 
profile (If It would make a good Illustration) 

6. Decisions based on measurement (e.g., prescribe 
Individualized Instructional program, group 
students, release from program, other) 

D. Special provision for teacher planning (e.g., regular 
meetings to plan each Individual's program, other) . IPD 9.06 



V. 



SPECIFIC EXAMPLES OF INSTRUCTIONAL PROCEDURES 



IPD REFERENCE 
IPD 9.00 



Instructions 

sections IV and V should be a valuable resource for a 
teacher Implementing the program at a new site. Important 
procedures should be described so clearly that a teacher 
new to the program could try to Imitate them. Draw on 
points under Sections III and IV and elaborate. (See 
AlDhaohonlcs and M.) 



VI. PARENT/COMMUNITY INVOLVEMENT (If applicable) 



IPD 9.08 



Instructions 

mere may not be enough to say about this aspect of the 
program to warrant a separate section. For example, In 
AH Day Kindergarten , a paragraph on parent Involvement 
is Included at the end of Section IV as an Incidental 
program feature. However, If this Is a well developed 
component, consider making It a separate section. Cover 
the following points at least: 



A. Specific nature of Involvement 

B. Concrete results 

C. Measures taken to secure their support 



VII. EQUIPMENT, MATERIALS, AND FACILITIES 



Instructions 

it may be more appropriate to cover facilities (C) In sec- 
tion IV without separating your description from the dlscusj- 
slon of activities. See All Day Kindergarten for an exam- 
ple of how A and B can be covered to maxe this a very use- 
ful section for the teachers In your audience. 



A. Major Items of equipment and material required for 

program IPO 11 .01 

B. Purpose/ procedures for use of each key Item (related 

to specific Instructional events) . IPD 11.02 

C. Description, special construction/alterations 

required for program facilities IPD 10.01, 10.02 



VIII. PROVISIONS FOR TEACHERS TO MODIFY FEATURES OF THE PROGRAM IPD 9.06 



E-22 
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IPD REFERENCE 



IX. QUOTED SOURCES (documents) 



IPD 1.02 



Instructions 

using correct bibliographic format, give entries for any 
documents from whicti you lifted material to Include In 
this Handbook . Lifting means quoting words or using pro- 
gram material as illustrative plates. 



X. FOR FURTHER INFORMATION (program contacts) IPD 1.01 

Instructions 

indicate for one or two contacts which you have cleared 
with program staff. See Alphaphonlcs and All Day Kinder- 
garten for examples of this section. 



E-23 
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OUTLINE FOR THE 
PROGRAM MANAGEMENT HANDBOOK 



Audience ; This handbook Is prepared for people with the authority to make or to 
reconmend changes In existing educational practices. Superintendents 
of Instruction, reading supervisors and coordinators, and principals 
would be Included In this group. 

Format ; Headings In ALL CAPS below have to be used In the handbook. Use sub- 
headings which best organize the material you present. REFER TO HAND- 
BOOKS FOR ALPHAPHONICS AND FOR THE ALL DAY KINDERGARTEN for additional 
format Ideas unt11 more handbooks are written. 

IPD REFERENCE 

I. PROGRAM OVERVIEW 



Instructions 

\jUnout omitting Important Information, shorten the same 
Program Overview you prepared for the Instructional Hand- 
book to 200-300 words. This Is about T^rpTui~pagii~o? 
letter ^othlc (large) type. 



11. HOW THE PROGRAM DEVELOPED 



Instructions 

Report Information which Is Important, not just nice, for 
the audience to know. See this section In the Program 
Management Handbook for All Day Kindergarten and Alpha - 
phonics for the sort of information that is considered 
essential background. 



A. Initial stimulus for program (effort of key person, 
outgrowth of earlier effort) IPD 6.02 

B. Needs assessment (If applicable) IPD 6.03 

1. By whom 

2. Methods/Instruments 

3. Needs selected for program (priorities) 

4. Who makes these decisions 

5. Useful materials for package (Items that would 
make a good Illustration) 

C. Planning process (before program began) ...... IPD 6.04 

1. Major steps 

2. Key persons/roles In major planning tasks 

3. Useful materials for package (e.g., flow charts, 
time lines, calendars, or other Items which could 
be reproduced as Illustrations) 
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IPD REFERENCE 



D. Implementation process ........ IPD 6.05 

1. Major steps 

2. Key persons/roles In major Implementation tasks 

3. Effective/Ineffective procedures 

4. Useful materials for package (e.g., C3 above) 

E. Changes In program from original plan IPD 6.06 

(If applicable, this may fit better under D unless 

changes were very substantial and deserve discus- 
sion In a separate section,) 

1. How (changes In goals, activities, staff, etc.) 

2. Why (recruitment problems, community pressure, 
budget cuts) 

F. Program's growth since It began IPD 6.07 



III. HOW THE PROGRAM IS STAFFED 



Inservlce 

Reproduce the chart from the PDF except for the column on 
Qualifications. Number the X's In the chart. Below the 
chart, use corresponding numbers for paragraphs which ex- 
plain the X's. If there Is a lot of Information on Inser- 
vlce training which calls for a separate section on that 
component, try the format Illustrated In the Program Man- 
agement Handbook for the All Day Kindergarten . 



A. Tabular presentation of type, number time devoted to 
program, and special requirements re role or Inservlce 



training IPD 7.01 

(PDF) 

B. Problems In recruiting or maintaining staff 
(Incentives to attract staff, teachers allowed to 

leave program) IPD 7.03 

C. Inservlce training In detail (may call for separate 

section) IPD 7.02 



1. Specific objectives 

2. Key persons/roles In planning/conducting 

3. When and where (sample schedule) 

4. Instructional methods/activities 

5. Assessment of staff proficiency In skills taught 
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IPD REFERENCE 



HOW THE PROGRAM IS MANAGED 



Instruction 

There will be wide variation among programs In this sec- 
tion. Two extremes are exemplified by the Program Manage- 
ment Handbooks for Alphaphonlcs and for the All pay Kinder- 
garten ; however. In each case there was Important Infor- 
mation which could be presented for A, B, and C. 



A. Chain of command IPD 8.01 

1. Protocol for who tells whom what (chart If appro- 
priate) 

2. Cay-to-day situations In which 'itris routine Is 
modified to facilitate program management 

3. Prucedures for decision-making re unsolicited 
advice from teachers, students, and parents 

B. Periodic review of programs (process evaluation) . . IPD 8.02 

1 . Program procedures assessed 

2. Who does what (key persons/roles) 

3. Method/ Instruments used 

4. Measurement schedule/frequency 

5. Major decisions made on basis of review 

6. Who makes these decisions 

C. Opinions of program staff re most essential management 
techniques for replicating programs (Indicate Identity 

of person expressing the opinion) IPD 8.03 



IPD REFERENCE 

V. MOW THE PROGRAM IS BUDGETED 



Instructions 

The IPD COS? charts ; You may or may not have been able to 

complete the top of the charts, but 
certainly you were able to get some 
Inkling of how the program was 
budgeted by getting Information for 
the rest of the chart. A special 
case Is the program which Is paid 
for out of thft regular district 
budget. See below. 

Presenting cost data ; Present whatever Information you 

rould obtain In a way which will help 
the reader budget the program In his 
own situation. If the kind of Infor- 
mation you got can be presented under 
cost categories, use the ones in the 
charts plus additional ones you need. 
Refer to the format for the All Day 
Kindergarten cost section If you 
were able to make use of the IPD 
charts. If your program did not 
require funds beyond the money pro- 
vided by the regular school program. 
Just talk about budgeting for neces- 
sary program supplies or activities. 

See PLC for assistance on this sec- 
tion 1f you want to. 



A. Start-up cost IPD 12.03 

1. Give cost data at top of IPD chart If program 
gave It to you 

2. Interpret cost data via supplementary Information 
you got In filling ouTVest of chart and In answers 
to questions on IPD page 15 

3. }f 1 and/or 2 are not possible, summarize what you 
cid get (e.g., major Items of expense which should 
be considered In budgeting the first year of the 
program) 

B. Continuation cost IPD 12.04 

1 . Same as Al 

2. Same as A2, except refer to questions on IPD page 17 

3. Same as A3, but hopefully there will be something 
to say under each mi.;)r cost category when you do 
B2 
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IPD REFERENCE 



C. 



Sunmary 



(PDF) 



VII. 



VIII. 



If yours Is a fairly long cost section (see All Day 
Kindergarten) . pull together the most ImportantpoTnts 
the reader should keep In mind In budgeting for this 
program. Refer back to the cost section of the PDF 
which may contain Information not yet covered In vie 
preceding cost paragraphs. The Summary could go at 
either end of the cost section. 



HOW THE PROGRAM IS EVALUATED 



Instructions 

Get a validator to write this according to the format used 
In the Program Management Handbooks for Alphaphonlcs and 
the All uay Kindergarten , in general , this section covers 
the following: tests used, comparison methods, and data 
analysis. The style should match your style . 



MODIFICATIONS PLANNED FOR FUTURE IPD 13.00 



Instructions 

Any possibly Important future modification—of staff util- 
ization, program design, classroom activities, Inservlce 
training— should be mentioned In this section. 



QUOTED SOURCES (documents) IPD 1.02 



Instructions 

Using correct bibliographic format, give entries for any 
documents from which you lifted material to Include In 
this Handbook . Lifting means quoting words or using pro- 
gram maIerTaTas"TTrusirat1ve plates. 



FOR FURTHER INFORMATION (program contacts) IPD 1.01 



Instructions 

Indicate for one or two contacts which you have cleared 
with program staff. See Alphaphonlcs and All Day Kinder- 
garten for examples of this section. 
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APPENDIX F 



Sample Program Description for the 
Catalog of Reading Programs 
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Sample Program Description for the 
Catalog of treading Programs 



ENRICHED AND EXTENDED SCHOOL YEAR PROGRAM 
EAST CLEVELAND; OHIO 



PROGRAM SIZE AND 
TARGET POPULATION 



YEAR STARTED 



STAFF 



About 1.500 children from 5 elementary schools are In this program. Almost 
100'^ are black and come from an inner-city community in which family income 
is low. with 50% of the school children from welfare families. 

The program began in 1971. 

The administrative staff consists of a program coordinator, a coordinator of 
supportive services, a budgetary manygcr, and an evaluation adviser. There arc 60 
teachers, assisted by 10 paraprofcssionals. all of whom participate in ongoing 
inservice training sessions. Thirty specialists in the fields of art. mu.sic. drama, 
science, history, and health provide additional instruction. 



MAJOR FEATURES Children selected for this program ihow a deficit in reading achievement 

eiiuivalent to I years or more. The program focuses on strengthening reading, 
vocabulary, and comnumication skills through the addition of nontraditional 
settings for instruction and the extension of the school year through July. The 
extended year is organized into six 6-week segments. Hach segment includes 4 
weeks of traditional classroom instruction. I week of special instruction at one 
of several community cultural institutions, and I week of vacation. Scheduling 
varies, but all classes spend 5 days, not necessari'y consecutively, within each 
6-week period at one of the institutions. The teacher and specialist from each 
cultural center use part of the I -week vacation time to plan activities for the 
following instructional period. The activities are designed to enrich classroom 
experiences while also teaching vocabulary and oral communication skills. For 
these 5 days, the institution specialist becomes the teacher, and the teacher 
becomes an active participant in the program. At the end of the 5 days, both 
teachers meet for a feedback session. Participating cultural centers are the 
Cleveland Art Museum, the Music Settlement House. Fairmount Center for 
Creative and Performing Arts, the Cleveland Health Museum. Hiram College 
(Center of Biological Studies). Western Reserve Historical Society. Cleveland 
Zoo, the Cleveland Natural Science Museum. Resident Camp, and Karamu House 
Theatre. Integrated into the regular classroom curriculum is time for the children 
to write about their experiences at these institutions. They are encouraged to use 
their new vocabulary in diary entries, and learning to read by writing augments 
their regular reading lessons. In addition to planning activities, teachers schedule 
parent conferences and tutor students during the 1-week vacation periods. 

FACILITIES, This program uses the facilities of the cultural centers already mentioned. This 
MATERIALS, includes classroom, gallery, and workshop space and supplies and equipment for 
EQUIPMENT special projects. 



COST The total cost of instructional materials for a class of 30 is $6,000. In addition. 

each cultural center is paid an average of $15 per pupil per week of instruction. 
The average, annual per-pupil cost for the regular district program is $1,000. The 
annual per-pupil cost of this program exceeds that by $200. 
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